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Preface 


About  This  Document 

Executive  Order  12866,  as  supplemented  by  Executive  Order  13563,  requires  federal  agencies 
to  evaluate  the  benefits,  costs,  and  other  impacts  of  major  regulations  prior  to  promulgation. 
For  regulations  intended  to  confer  benefits  under  circumstances  of  extreme  uncertainty,  such 
as  commonly  arise  in  the  context  of  homeland  security,  this  requirement  has  proven  especially 
challenging.  This  document  is  based  on  a  workshop  on  this  topic  that  was  conducted  jointly 
by  the  RAND  Corporation  and  Industrial  Economics,  Incorporated,  on  November  29,  2011, 
titled  “Assessing  the  Benefits  of  U.S.  Customs  and  Border  Protection  Regulatory  Actions  to 
Reduce  Terrorism  Risks.” 

The  workshop,  which  was  sponsored  by  U.S.  Customs  and  Border  Protection,  brought 
together  leading  experts  in  the  field  of  regulatory  analysis  and  terrorism  risk.  The  objective  of 
the  workshop  was  to  examine  alternative  approaches  for  estimating  the  benefits  of  regulations 
designed  to  reduce  the  risks  of  terrorist  attacks  on  U.S.  soil.  The  observations  presented  in  this 
document  are  not  RAND’s  but  reflect  those  of  the  workshop  participants.  This  document 
should  be  of  interest  to  those  in  the  homeland  security  community  and  in  other  policy  com¬ 
munities  who  are  concerned  with  the  conduct  of  benefit-cost  analysis  under  circumstances  of 
extreme  uncertainty.  Readers  may  also  be  interested  in  related  RAND  research  on  terrorism 
risk  analysis  and  management,  including 

•  Brian  A.  Jackson,  Edward  W.  Chan,  and  Tom  LaTourrette,  “Assessing  the  Security  Ben¬ 
efits  of  a  Trusted  Traveler  Program  in  the  Presence  of  Attempted  Attacker  Exploitation 
and  Compromise,”  WR-855-RC,  Santa  Monica,  Calif:  RAND  Corporation,  2011. 

•  Andrew  R.  Morral  and  Brian  A.  Jackson,  Understanding  the  Role  of  Deterrence  in 
Counterterrorism  Security,  OP-281-RC,  Santa  Monica,  Calif:  RAND  Corporation, 

2009. 

•  Henry  H.  Willis  and  Tom  LaTourrette,  “Using  Probabilistic  Terrorism  Risk  Modeling  for 
Regulatory  Benefit-Cost  Analysis:  Application  to  the  Western  Hemisphere  Travel  Initia¬ 
tive  Implemented  in  the  Land  Environment,”  Risk  Analysis,  Vol.  28,  No.  2,  April  2008, 
pp.  325-339. 

The  RAND  Homeland  Security  and  Defense  Center 

This  research  was  conducted  in  the  RAND  Homeland  Security  and  Defense  Center,  which 
conducts  analysis  to  prepare  and  protect  communities  and  critical  infrastructure  from  natural 
disasters  and  terrorism.  Center  projects  examine  a  wide  range  of  risk  management  problems, 
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including  coastal  and  border  security,  emergency  preparedness  and  response,  defense  support 
to  civil  authorities,  transportation  security,  domestic  intelligence,  technology  acquisition,  and 
related  topics.  Center  clients  include  the  Department  of  Homeland  Security,  the  Department 
of  Defense,  the  Department  of  Justice,  and  other  organizations  charged  with  security  and 
disaster  preparedness,  response,  and  recovery.  The  Homeland  Security  and  Defense  Center  is 
a  joint  center  of  the  RAND  National  Security  Research  Division  and  RAND  Infrastructure, 
Safety,  and  Environment. 

Information  about  the  Homeland  Security  and  Defense  Center  is  available  online  (http:// 
www.rand.org/multi/homeland-security-and-defense/).  Inquiries  about  homeland  security 
research  projects  should  be  sent  to: 

Andrew  Morral,  Director 
Homeland  Security  and  Defense  Center 
RAND  Corporation 
1200  South  Hayes  Street 
Arlington,  VA  22202-5050 
703-413-1100,  x5119 
Andrew_Morral@rand.org 
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Summary 


Executive  Order  (E.O.)  12866,  as  supplemented  by  E.O.  13563,  requires  federal  agencies  to 
evaluate  the  benefits,  costs,  and  other  impacts  of  major  regulations  prior  to  promulgation.  For 
regulations  intended  to  confer  benefits  under  circumstances  of  extreme  uncertainty,  such  as 
commonly  arise  in  the  context  of  homeland  security,  this  requirement  has  proven  especially 
challenging.  To  assist  U.S.  Customs  and  Border  Protection  (CBP),  a  key  component  of  the 
U.S.  Department  of  Homeland  Security  (DHS),  in  meeting  these  challenges,  the  RAND  Cor¬ 
poration  and  Industrial  Economics,  Incorporated,  conducted  a  workshop  in  November  2011, 
titled  “Assessing  the  Benefits  of  U.S.  Customs  and  Border  Protection  Regulatory  Actions  to 
Reduce  Terrorism  Risks,”  that  drew  together  leading  experts  in  the  field  of  regulatory  analysis 
and  terrorism  risk. 

Most  of  the  ideas  that  emerged  from  the  workshop  pertained  to  either  qualitative  mod¬ 
eling,  quantitative  modeling,  or  data  collection.  Participants  also  considered  a  set  of  cross¬ 
cutting  issues,  specifically  those  of  analytical  capacity,  transparency,  and  presentation.  A  distil¬ 
lation  and  synthesis  of  the  emerging  ideas,  recurring  themes,  and  (albeit  few  in  number)  points 
of  contention  among  the  participants  suggested  several  recommendations  to  assist  CBP  in 
meeting  the  challenges  of  improving  the  benefit-cost  analysis  of  terrorism  security  regulations: 

•  Aim  for  break-even  work  that  is  more  descriptive.  Comprehensive  or  “full”  benefit- 
cost  analysis  is  likely  neither  attainable  nor  desirable,  given  the  extent  of  uncertainty  and 
related  analytical  challenges,  but  break-even  analysis  can  be  used  to  explore  the  implica¬ 
tions  of  uncertainty  and  describe  conditions  under  which  benefits  could  exceed  costs  for 
particular  regulations. 

•  Incorporate  ‘‘storytelling”  into  regulatory  analysis  and  consider  adopting  logic 
modeling  as  part  of  the  regulatory  development  process.  Logic  models  can  be  used 
to  support  benefit-cost  analysis,  break-even  analysis,  and  regulatory  development,  more 
generally,  by  helping  to  identify  and  articulate  plausible  links  between  regulatory  actions 
and  outcomes. 

•  Strengthen  internal  modeling  capacity  and  leverage  existing  risk  assessment  meth¬ 
ods,  when  possible,  including  DHS/Office  of  Risk  Management  and  Analysis  (RMA) 
and  computable  general  equilibrium  (CGE)  models.  A  varied,  multidisciplinary  tool¬ 
kit  can  be  used  to  assess  big  effects  and  shed  light  on  uncertainty.  For  example,  CGE 
models  can  be  used  to  calculate  welfare  effects;  DHS/RMAs  risk  assessment  models  to 
estimate  risk;  and,  as  already  noted,  logic  models  to  “tell  the  story”  of  a  regulatory  action 
and  its  effects. 

•  Develop  and  adopt  DHS-wide  standards  for  reliable  expert  elicitation.  Learning  more 
about  best  practices — to  better  understand  the  technique  s  strengths,  weaknesses,  and 
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applicability  to  benefit-cost  analysis — and  establishing  criteria  based  on  those  practices 
would  help  to  ensure  that  expert  elicitation  throughout  DHS  produces  reliable  results. 

•  Improve  basic  science  in  potentially  important  but  under-explored  areas  of  terror¬ 
ism  consequences,  including  behavioral  responses,  mental  health,  and  fear.  Not 

only  does  fear  have  behavioral  implications  that  can  impact  the  probability  or  magnitude 
of  effects,  it  may  have  direct  bearing  on  an  individuals  mental  health,  well-being,  and 
happiness. 

•  Incorporate  pilot  studies  and  other  natural  experiments  into  regulatory  design. 

Regulatory  rollouts  that  create  natural  experiments,  e.g.,  through  pilot  programs,  interim 
final  rules,  or  phased  implementation,  may  yield  data  and  enable  analysis  that  provides 
insight  to  the  effects  of  final  rules. 

•  Improve  transparency.  It  may  be  possible  to  improve  the  transparency  of  benefit-cost 
analysis,  e.g.,  by  sharing  information  when  doing  so  would  not  jeopardize  security  and 
expanding  verification  and  validation. 
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SECTION  1 


Introduction 


Executive  Order  (E.O.)  12866  (Clinton,  1993),  as  supplemented  by  E.O.  13563  (Obama, 
2011),  requires  federal  agencies  to  evaluate  the  benefits,  costs,  and  other  impacts  of  major 
regulations  prior  to  promulgation.  For  regulations  intended  to  confer  benefits  under  circum¬ 
stances  of  extreme  uncertainty,  such  as  commonly  arise  in  the  context  of  homeland  security, 
this  requirement  has  proven  especially  challenging.  To  assist  U.S.  Customs  and  Border  Protec¬ 
tion  (CBP),  a  key  component  of  the  U.S.  Department  of  Homeland  Security  (DHS),  in  meet¬ 
ing  these  challenges,  the  RAND  Corporation  and  Industrial  Economics,  Incorporated  (lEc), 
conducted  a  workshop,  titled  “Assessing  the  Benefits  of  U.S.  Customs  and  Border  Protection 
Regulatory  Actions  to  Reduce  Terrorism  Risks,”  that  drew  together  leading  experts  in  the  field 
of  regulatory  analysis  and  terrorism  risk.  The  objective  of  the  workshop  was  to  examine  alter¬ 
native  approaches  for  estimating  the  benefits  of  CBP  regulations  designed  to  reduce  the  risks 
of  terrorist  attacks  on  U.S.  soil. 


Implementing  Guidance 

In  its  implementing  guidance  for  E.O.  12866  and  related  authorities  (“Circular  A-4”),  the  U.S. 
Office  of  Management  and  Budget  (OMB,  2003,  pp.  2-3)  stipulates  that  an  evaluation  of  ben¬ 
efits  and  costs  should  do  the  following^ 

1.  Explain  how  the  actions  required  by  the  rule  are  linked  to  the  expected  benefits.  Sepa¬ 
rate  analyses  should  be  done  for  each  of  the  alternatives. 

2.  Identify  a  baseline.  Benefits  and  costs  are  defined  in  comparison  with  a  clearly  stated 
alternative.  This  normally  will  be  a  “no  action”  baseline:  what  the  world  will  be  like  if 
the  proposed  rule  is  not  adopted.  Comparisons  to  a  “next  best”  alternative  are  also  espe¬ 
cially  useful. 

3.  Identify  the  expected  undesirable  side  effects  and  ancillary  benefits  of  the  proposed 
regulatory  action  and  the  alternatives.  These  should  be  added  to  the  direct  benefits  and 
costs  as  appropriate. 


^  As  explained  by  OMB  in  the  preamble  to  the  circular,  “This  Circular  provides  the  Office  of  Management  and  Budget’s 
(OMB’s)  guidance  to  Federal  agencies  on  the  development  of  regulatory  analysis  as  required  under  Section  6(a)(3)(c)  of 
Executive  Order  12866,  ‘Regulatory  Planning  and  Review,’  the  Regulatory  Right-to-Know  Act,  and  a  variety  of  related 
authorities.” 
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OMB  (2003,  p.  2)  affirms  that  the  motivation  for  the  analysis  is  to  learn  whether  the 
benefits  of  an  action  are  likely  to  justify  the  costs  or  to  discover  which  of  various  possible  alter¬ 
natives  would  be  the  most  cost-effective,  but  OMB  also  notes  that  efficiency  might  not  be  the 
only  or  the  overriding  public  policy  objective. 

OMB  asks  agencies  undertaking  a  benefit-cost  analysis  to  quantify  anticipated  benefits 
and  costs  to  the  extent  possible  but  recognizes  that  quantification,  particularly  monetization, 
may  not  be  feasible  in  all  cases.  In  instances  in  which  it  is  not  possible  to  monetize  benefits  or 
costs,  an  agency  may  calculate  them  in  terms  of  physical  units,  e.g.,  stream  miles  of  improved 
water  quality  or  increases  in  game  fish  populations  (OMB,  2003,  p.  27);  when  physical  units 
are  unattainable,  an  agency  may,  instead,  provide  alternative  information  to  support  the  regu¬ 
latory  process: 

Even  when  a  benefit  or  cost  cannot  be  expressed  in  monetary  units,  you  should  still  try  to 
measure  it  in  terms  of  its  physical  units.  If  it  is  not  possible  to  measure  the  physical  units, 
you  should  still  describe  the  benefit  or  cost  qualitatively.^  (p.  10) 

OMB  also  emphasizes  the  importance  of  reporting  “transfers”  separately  and,  thus,  iden¬ 
tifying  the  net  effects  of  a  proposed  regulation  on  “aggregate  social  welfare.”^  According  to 

OMB  (2003,  p.  46), 

You  [the  agencies  and  establishments  conducting  the  benefit-cost  analysis]  should  report 
transfers  separately  and  avoid  the  misclassification  of  transfer  payments  as  benefits  or  costs. 
Transfers  occur  when  wealth  or  income  is  redistributed  without  any  direct  change  in  aggre¬ 
gate  social  welfare.  To  the  extent  that  regulatory  outputs  reflect  transfers  rather  than  net 
welfare  gains  to  society,  you  should  identify  them  as  transfers  rather  than  benefits  or  costs. 


Analytical  Challenges 

CBP,  like  other  DHS  components,  faces  considerable  challenges  in  conducting  benefit-cost 
analyses  for  regulations  designed  to  reduce  the  risks  of  terrorist  attacks  on  U.S.  soil.^  Whereas 
the  anticipated  costs  might  be  reasonably  well  defined,  the  benefits,  which  hinge  on  the  avoid¬ 
ance  of  damages  under  extremely  uncertain  conditions,  tend  to  be  difficult  to  measure.  Inso¬ 
much  as  terrorism  security  regulations  are  intended  to  deter  or  otherwise  prevent  terrorist 
attacks,  the  benefits  come  from  the  averted  damages  (DHS,  2009;  Willis  et  ah,  2005;  Willis 
and  LaTourrette,  2008).  Estimating  those  benefits  would  typically  involve  evaluating  soci¬ 
ety’s  willingness  to  pay  to  avoid  terrorism-related  damages  or,  lacking  information  on  willing¬ 
ness  to  pay,  estimating  the  expected  change  in  terrorism-related  damages  that  would  result 


^  OMB  also  allows  the  possibility  of  cost-effectiveness  analysis  (p.  10)  and  “threshold”  or  “break-even”  analysis  (p.  2).  See 
the  later  discussion  of  break-even  analysis,  below. 

^  In  economic  parlance,  a  measure  of  “aggregate  social  welfare”  is  the  overall  economic  “surplus”  of  consumers  and  pro¬ 
ducers,  itself  a  measure  of  economic  well-being.  Consumer  surplus  measures  aggregate  net  benefits  to  consumers;  that  is, 
the  difference  between  what  a  consumer  is  willing  to  pay  for  a  quantity  of  a  good  or  service  and  what  he  or  she  actually  pays, 
summed  over  all  consumers.  Likewise,  producer  surplus  measures  aggregate  net  benefits  to  producers;  that  is,  the  difference 
between  a  firm’s  threshold  price  and  the  actual  price  of  the  good  or  service,  summed  over  all  firms. 

For  purposes  of  this  report,  we  refer  to  these  types  of  regulations  as  “terrorism  security  regulations.” 
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from  implementation  of  the  new  regulation  (Smith,  Mansfield,  and  Clayton,  2008;  Willis  and 
LaTourrette,  2008).  Both  approaches — assessing  willingness  to  pay  or  the  expected  change 
in  damages — require  an  explicit  characterization  of  the  underlying  risk  of  a  terrorist  attack. 
However,  benefit-cost  analysis  has  yet  to  overcome  two  analytical  challenges  associated  with 
estimating  the  baseline  level  of  terrorism  risk  and  the  anticipated  effects  of  proposed  security 
measures  on  that  risk:  a  lack  of  data  with  which  to  estimate  terrorism  risks  and  an  inability  to 
anticipate  how  terrorists  will  adapt  to  changes  in  the  security  environment  (Ezell  et  ah,  2010; 
Merrick  and  Parnell,  2011;  National  Research  Council,  2010). 

Relevant,  reliable,  and  accurate  data  for  evaluating  either  the  baseline  risk  posed  by  terror¬ 
ists  or  the  anticipated  change  in  risk  resulting  from  a  regulation  are  scant  for  two  reasons.  First, 
there  have  been  few  terrorist  attacks  or  attempted  attacks  in  the  United  States  or  in  comparable 
contexts.  Second,  the  few  attempted  attacks  have  varied  greatly  in  nature,  thus  providing  little 
data  with  which  to  characterize  even  those  scenarios  drawn  from  actual  experience.  As  a  result, 
the  record  of  terrorism  does  not  allow  for  extrapolation  of  risk  estimates  that  consider  the  dif¬ 
ferent  ways  terrorists  might  attack  in  the  future.  Even  when  careful  analysis  of  the  historical 
record  provides  a  means  to  estimate  the  baseline  risk,  it  is  difficult  to  know  whether  or  how 
much  that  risk  may  have  changed  because  of  some  deliberate  security  measures  adopted  under 
a  regulation  or  because  of  some  other  factors  whose  effects  cannot  be  isolated. 

Conventional  treatments  of  risk,  such  as  those  focusing  on  probability  and  consequence 
or  on  threat,  vulnerability,  and  consequence,  also  fall  short,  in  part  because  terrorist  adversaries 
are  adaptive  and  the  environment  is  dynamic  (Ezell  et  al.,  2010;  Willis  et  al.,  2005).  Changes 
in  circumstances  might  lead  an  adversary  to  adopt  a  new  strategy  to  achieve  the  same  end  or  to 
alter  the  overall  intensity  of  its  efforts.  When  an  analysis  is  possible,  concerns  about  releasing 
or  “backing  out”  security-sensitive  information  have  sometimes  constrained  federal  agencies’ 
ability  to  carry  out  assessments  and  to  vet  models  or  publish  findings. 

In  response  to  these  conceptual  and  practical  challenges,  CBP  has  been  using  a  “break¬ 
even”  approach  to  conduct  regulatory  impact  analysis.  Rather  than  attempt  to  estimate  ben¬ 
efits  directly,  break-even  analysis  identifies  the  conditions  necessary  for  the  benefits  of  the 
regulation  to  exceed  the  costs: 

It  will  not  always  be  possible  to  express  in  monetary  units  all  of  the  important  benefits  and 

costs . If  the  non-quantified  benefits  and  costs  are  likely  to  be  important,  you  should 

carry  out  a  “threshold”  analysis  to  evaluate  their  significance.  Threshold  or  “break-even” 
analysis  answers  the  question,  “How  small  could  the  value  of  the  non-quantified  benefits  be 
(or  how  large  would  the  value  of  the  non-quantified  costs  need  to  be)  before  the  rule  would 
yield  zero  net  benefits?”  (OMB,  2003,  p.  2) 

The  relevant  conditions,  such  as  baseline  risk  and  risk  reduction,  can  then  be  evaluated  for 
feasibility.  The  method  does  not,  however,  project  whether  the  regulation  will  achieve  neces¬ 
sary  risk  reductions.  Although  offered  as  an  option  in  Circular  A-4,  CBP  has  expressed  interest 
in  identifying  novel  approaches  to  valuing  benefits  and,  potentially,  moving  beyond  break-even 
analysis. 
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Workshop  Objectives,  Charge,  and  Format 

To  assist  CBP  in  meeting  the  challenges  of  assessing  the  benefits  of  terrorism  security  regula¬ 
tions,  RAND  and  lEc  conducted  a  day-long  workshop  in  November  2011,  titled  “Assessing 
the  Benefits  of  U.S.  Customs  and  Border  Protection  Regulatory  Actions  to  Reduce  Terrorism 
Risks.”  The  workshop  provided  an  opportunity  for  leading  experts  in  the  fields  of  regulatory 
analysis  and  terrorism  risk  to  identify  and  discuss  alternative  approaches  for  valuing  the  ben¬ 
efits  of  CBP  regulations,  specifically  those  designed  to  reduce  the  risk  of  terrorist  attacks. ^ 

Four  participants  were  asked  to  prepare  presentations  describing  possible  short-  and  long¬ 
term  approaches  to  improving  the  analysis  of  terrorism  security  regulations  and  to  provide 
advice  and  guidance  related  to  developing  and  implementing  these  approaches.  These  are 
included  in  Appendix  C.  To  help  focus  the  presentations  and  to  facilitate  comparisons  in  a 
common  context,  participants  were  asked  to  discuss  how  their  approaches  would  be  applied  to  a 
rule  recently  implemented  by  CBP  and  the  U.S.  Department  of  State:  the  Western  Hemisphere 
Travel  Initiative  (WHTI).  WHTI  tightened  and  streamlined  the  documentation  requirements 
for  travelers  entering  the  United  States  from  certain  countries  in  the  Western  Hemisphere, 
including  Canada  and  Mexico.  It  was  implemented  in  the  air  environment  in  2006  and  the 
land  and  sea  environments  in  2009.  Prior  to  WHTI,  regulations  permitted  U.S.  citizens  and 
nonimmigrant  aliens  from  Canada,  Bermuda,  and  Mexico  to  enter  the  United  States  without 
a  passport.  Subsequent  to  WHTI,  regulations  require  that  nearly  all  entrants  present  either 
a  passport  book,  passport  card  with  vicinity-read  radio-frequency  identification  chip,  CBP 
trusted  traveler  card,  DHS  Enhanced  Driver  s  License,  or  Merchant  Mariner  Document.  See 
lEc  (March  2008). 

Using  WHTI  as  a  point  of  reference,  the  presentations  and  surrounding  discussions 
addressed  questions  about 

1.  differences  between  the  proposed  approaches  and  other  types  of  regulatory  benefits 
analyses 

2.  prior  analyses  (regulatory  or  other)  from  which  methods  or  findings  could  be  applied 

3.  any  models  that  could  be  used  in  analyses 

4.  anticipated  software  or  other  computing  requirements 

5.  anticipated  data  needs  and  data  collection  options 

6.  consistency  with  OMB  guidance  for  estimating  the  benefits  of  regulations. 

The  emphasis  of  the  workshop  was  on  benefits  related  to  terrorism  security;  however,  par¬ 
ticipants  were  also  encouraged  to  address  when  and  how  it  would  be  appropriate  to  consider 
ancillary  benefits.  As  CBP  faces  far  fewer  challenges  in  estimating  potential  costs,  the  costs 
of  the  sample  rule,  including  indirect  effects,  such  as  increased  wait  times  or  business  losses 
resulting  from  decreased  border  crossings,  were  explicitly  excluded  from  the  workshop.^ 

The  remainder  of  this  report  of  the  workshop  proceedings  draws  together  key  elements  of 
the  presentations,  discussions,  and  recommendations  of  the  group.  It  is  not  intended  to  serve 
as  a  transcript  of  the  workshop;  rather,  it  is  intended  to  provide  a  distillation  and  synthesis  of 
emerging  ideas,  recurring  themes,  and,  though  limited  in  number,  noteworthy  points  of  con- 


^  Appendix  A  contains  a  complete  list  of  participants;  Appendix  B  provides  the  agenda  for  the  workshop. 
^  Nevertheless,  some  participants  did  choose  to  address  unintended  consequences  as  a  secondary  issue. 
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tention  among  the  participants/  This  report  does  not  attribute  remarks  to  particular  individu¬ 
als,  except  as  the  authors  of  the  presentations  that  they  prepared  in  advance  of  the  workshop. 


^  Sections  2,  3,  and  4  draw  information  solely  from  the  presentations  and  discussions  at  the  workshop,  unless  specifically 
noted  otherwise. 


SECTION  2 


Emerging  Ideas  and  Recurring  Themes 


Although  CBP  expressed  interest  in  identifying  novel  approaches  to  valuing  benefits  and, 
potentially,  moving  beyond  break-even  analysis,  participants  did  not  issue  a  call  for  a  new  ana¬ 
lytical  paradigm;  instead,  they  identified  opportunities  to  augment  existing  approaches  and 
create  a  stronger  analytical  foundation  for  future  assessments.  The  discussions  lent  support  to 
benefit-cost  analysis  as  a  useful  analytical  construct  in  the  regulatory  process  and  break-even 
analysis  as  an  important  means  of  confronting  uncertainty.^  Given  the  potential  for  feedback, 
the  completion  and  review  of  a  benefit-cost  analysis  can  provide  agencies,  the  Congress,  and 
the  public  with  a  means  of  iterating  toward  better  policies  and  practices.  For  example,  the  ben¬ 
efit-cost  analysis  of  a  congressionally  mandated  regulation  could  shed  light  on  an  unintended 
cost,  which  might  then  lead  to  a  change  in  law,  which  might,  in  turn,  result  in  the  adoption  of 
a  less  costly  but  still  effective  regulatory  strategy.  ^  On  balance,  the  discussions  supported  what 
one  participant  termed  “the  traditional  role”  of  benefit-cost  analysis  as  a  descriptive  tool  rather 
than  a  normative  tool,  one  better  suited  to  informing  regulatory  decisions  than  to  prescribing 
them.  Participants  also  stressed  the  value  of  benefit-cost  analysis  as  a  tool  for  explaining  the 
reasoning  behind  regulatory  decisions. 

Most  of  the  ideas  that  emerged  from  the  workshop  pertained  to  either  qualitative  mod¬ 
eling,  quantitative  modeling  (both  statistical  and  simulation),  or  data  collection,  potentially 
through  expert  elicitation  and  natural  experiments.  Throughout  the  day,  participants  spoke  to 
the  importance  of  focusing  on  the  “big”  or  main  effects  of  proposed  regulations — referring  to 
the  major  averted  damages  (e.g.,  death  and  injury,  property  losses,  other  substantial  economic 
losses^)  and  potentially  including  some  important  ancillary  effects  and  unintended  conse¬ 
quences.  With  regard  to  methodological  improvements,  participants  suggested  targeting  “low- 


^  See  Baxter’s  “Overview  of  Challenges”  in  Appendix  C  for  more  information  about  break-even  analysis.  In  discussing  the 
relationship  between  benefit-cost  analysis  and  break-even  analysis,  participants  noted  that  the  two  are  effectively  equivalent 
under  conditions  of  extreme  uncertainty:  in  particular,  if  the  probability  of  a  terroristic  attack  is  fundamentally  “unknow¬ 
able.”  The  two  approaches  make  use  of  the  same  information,  but  they  arrange  that  information  differently.  In  particular, 
break-even  analysis  treats  uncertain  parameters  as  variables  rather  than  as  known  parameters.  For  more  on  this  relationship, 
see  Hammitt’s  presentation  in  Appendix  C. 

^  Participants  noted  one  instance  in  which  a  cost  estimate  drove  a  change  in  law  and  another  in  which  a  cost  estimate 
drove  a  change  in  promulgation,  absent  a  change  in  law.  Regarding  the  latter,  they  identified  the  Importer  Security  Filing 
and  Additional  Carrier  Requirements  (“10+2”)  rule  (see  later  discussion)  as  a  case  in  which  the  benefit-cost  analysis  revealed 
uncertainty  over  outcomes,  including  costs,  and  led  to  a  change  in  promulgation.  CBP  chose  to  issue  an  interim  rule  to 
allow  time  to  develop  a  better  understanding  of  the  potential  effects  of  the  regulatory  action  and,  if  needed,  adjust  the  final 
rule  to  mitigate  undesirable  consequences. 

^  Examples  added  by  authors  for  illustrative  purposes. 
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hanging  fruit.”  Participants  also  considered  a  set  of  cross-cutting  issues,  specifically  those  of 
analytical  capacity,  transparency,  and  presentation. 


Qualitative  Modeling 

Participants  took  up  issues  of  qualitative  modeling  in  three  different  but  related  contexts: 

1.  in  setting  out  plausible  links  between  regulatory  actions  and  outcomes,  including  direct 
benefits,  ancillary  benefits,  and  unintended  consequences,  per  Circular  A-4  guidance 

2.  in  identifying  benefits  despite  the  limitations  of  quantification,  particularly  in  compara¬ 
tive  assessments  of  alternative  rules 

3.  in  valuing  more  or  better  information,  a  potential  positive  outcome  that  might  be  under¬ 
addressed  in  regulatory  analysis. 

Linking  Actions  to  Outcomes 

A  presentation  on  “logic  modeling”  introduced  a  qualitative  modeling  approach  that  is  rooted 
in  program  planning  and  evaluation  and  can  be  used  to  shed  light  on  the  logical  connections 
between  regulatory  actions  and  outcomes  (see  Greenfield  et  al.  in  Appendix  C).  The  approach 
could  enable  CBP  and  other  agencies  that  confront  extreme  uncertainty  to  link  rules,  actions, 
and  benefits;  it  can  be  used  to  create  a  roadmap  with  which  to  articulate  the  agency’s  program 
“story”  (i.e.,  the  what,  why,  and  who  of  a  program),  identify  how  a  proposed  regulation  would 
affect  that  story,  and  assess  resultant  changes  in  capabilities  and  outcomes  in  terms  of  the  story. 

Identifying  Benefits  Despite  Challenges 

Discussions  about  the  analytic  challenges  described  in  Section  1,  i.e.,  those  presented  by  uncer¬ 
tainty  and  pertaining  to  data  availability  and  model  validation,  suggested  a  more  central  role 
for  storytelling  in  benefit-cost  analysis. 

One  participant  referred  to  a  paper,  “The  Arithmetic  of  Arsenic”  (Sunstein,  2002),  by  the 
current  administrator  of  OMB’s  Office  of  Information  and  Regulatory  Affairs.  Recalling  a  key 
point  of  the  paper,  the  participant  asserted  that 

.  .  .  our  uncertainty  about  the  net  benefits  of  rules  is  typically  so  great  that  the  [quantita¬ 
tive]  analysis  has  very  little  value  in  distinguishing  among  candidate  rules  that  are  within 
the  feasible  set.  And  maybe  the  analysis  can  suggest  that  some  of  the  candidates  are  bad, 
but  the  theoretical  idea  of  finding  the  optimal  level  of  regulation  is  a  very  misleading  idea 
in  this  context. 

The  participant  concluded  that  one  could  still  reasonably  ask  for  a  “logical  story”  as 
to  how  a  proposed  rule  would  plausibly  reduce  terrorist  attacks  or  risks,  possibly  including 
some  discussion  of  countervailing  effects  (see  later  discussion  of  adaptive  behavior)  and  some 
quantification  of  the  costs  and  big  effects.  While  simple  analytical  models  might  prove  useful 
to  characterizing  those  effects,  discussions  during  the  workshop  raised  caution  about  efforts 
to  develop  complex  models,  citing  the  limited  value  of  putting  extensive  effort  into  detailed 
analysis  of  aspects  of  scenarios  given  uncertainty  about  whether  and  how  terrorism  will  evolve. 
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Valuing  Information 

The  implementation  of  a  new  regulation  can  generate  more  and  better  information,  which  can, 
in  turn,  have  value  to  both  the  government  and,  possibly,  to  individuals  and  institutions  out¬ 
side  the  government.  In  effect,  informational  gains  can  lead  to  improvements  in  “situational 
awareness.”  For  example,  tightening  and  streamlining  the  documentation  requirements  for 
travelers  entering  the  United  States  under  WHTI  might  help  to  shed  light  on  who  is  cross¬ 
ing  the  border,  when  they  are  crossing,  how  often  they  are  crossing,  and,  sometimes,  with 
whom  they  are  crossing.^  Participants  cited  the  Importer  Security  Filing  and  Additional  Car¬ 
rier  Requirements  rule  (commonly  known  as  “10+2”)5  as  an  example  of  a  regulatory  action 
leading  to  data  collection,  compilation,  and  sharing  that  might  confer  benefits  on  both  the 
government  and  private  sector.  Shippers  would  present  new  information  to  the  U.S.  govern¬ 
ment — in  some  instances,  “new”  only  to  the  government,  and  in  others,  “new”  to  the  shippers 
as  well — and  they  might  also  share  information  among  themselves.^ 

Participants  considered  whether  any  of  the  available  approaches  could  be  used  to  tease  out 
the  value  of  better  information,  hence  situational  awareness,  not  just  to  CBP,  but  to  other  DHS 
components,  other  agencies,  and  to  those  outside  the  government  who  are  affected  by  regula¬ 
tions.  If  a  regulation  is  expected  to  increase  situational  awareness  in  either  the  government  or 
the  private  sector,  could  any  of  the  approaches  be  used  to  value  the  effect?  The  logic  model 
appeared  to  hold  promise  as  a  tool  for  a  conceptual  exploration  of  those  gains.  One  could  use  a 
logic  model  to  trace  the  flow  of  information  through  a  program — eventually,  to  those  who  use 
the  information — and  to  assess  how  a  regulatory  action  would  affect  that  flow;  one  could  also 
use  the  model  to  articulate  the  role  of  information  in  linking  different  programs  and  agencies. 


Quantitative  Modeling 

Quantitative  modeling  provides  the  possibility  of  numerically  estimating  the  magnitude  of  the 
benefit  of  a  terrorism  security  regulation  and  uncovering  unanticipated  effects.  While  poten¬ 
tially  more  demanding  than  qualitative  modeling,  e.g.,  in  terms  of  needs  for  behavioral  under¬ 
standing  and  data,  it  can  provide  a  more  precise  description  of  the  effects  of  a  regulation  and 
thus  more  definitive  guidance  to  decisionmakers. 


^  This  example  derives  partly  from  a  workshop  presentation  (see  Greenfield  et  al.  in  Appendix  C),  from  conversations  with 
CBP  held  during  a  facility  visit  on  October  19,  2011,  and  from  the  description  of  the  intent  of  WHTI  in  lEc  (March  2008). 

^  CBP  (2009)  summarizes  the  terms  of  the  rule:  “Under  the  new  rule,  before  merchandise  arriving  by  [ocean]  vessel  can  be 
imported  into  the  United  States,  the  ‘Importer  Security  Filing  (ISF)  Importer,’  or  their  agent  (e.g.,  licensed  customs  broker), 
must  electronically  submit  certain  advance  cargo  information  to  CBP  in  the  form  of  an  Importer  Security  Filing.”  See  also 
lEc  (November  2008). 

^  CBP  (2009)  also  summarizes  the  intent  of  the  rule:  “The  information  submitted  in  Importer  Security  Filings  improves 
[CBP’s]  ability  to  identify  high-risk  shipments  in  order  to  prevent  smuggling  and  ensure  cargo  safety  and  security.”  lEc 
(November  2008,  p.  5-5)  addresses  potential  benefits  to  the  private  sector: 

Such  ancillary  benefits  might  include  long-term  improvements  in  supply-chain  efficiency  resulting  from  the  sharing  of 
higher  quality  information  in  a  more  timely  fashion  among  supply  chain  participants.  In  addition,  improved  visibility  into 
the  supply  chain  might  make  the  transportation  of  illegal  goods,  such  as  merchandise  fraudulently  advertised  as  being  the 
product  of  well-known  U.S.  companies  more  difficult. 

lEc  (p.  5-24)  posited  that  this  could  be  especially  valuable  to  importers  “who  currently  have  little  insight  into  the  process.” 
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Throughout  the  workshop,  discussions  of  quantitative  models  highlighted  the  extent  to 
which  CBP  and  other  agencies,  on  the  one  hand,  have  been  reliant  on  external — oftentimes 
proprietary — modeling  capacity  and,  on  the  other,  might  benefit  from  tapping  into  opera¬ 
tional  and  up-and-coming  internal  capacity.  These  discussions,  perhaps  more  so  than  those 
of  qualitative  methods,  also  highlighted  opportunities  to  build  on  advances  in  other  policy 
arenas.  In  contemplating  particular  quantitative  approaches,  participants  called  for  attention 
to  big  or  main  effects;  however,  they  did  not  dismiss  the  potential  significance  of  ancillary 
effects  or  unintended  consequences,  which  might  be  indirect  but  still  important. 

As  a  practical  matter,  participants  identified  four  opportunities  for  advancement,  in 
decreasing  order  of  perceived  tractability: 

1.  Make  use  of  emergent  DHS  risk  modeling  capacity. 

2.  Exploit  computable  general  equilibrium  (CGE)  models  to  identify  welfare  effects. 

3.  Better  characterize  behavior,  including  precautionary,  responsive,  and  adaptive  behavior. 

4.  Better  understand  the  consequences  of  fear  of  terrorism. 

Making  Use  of  New  Capabilities 

One  presentation,  that  of  Cheesebrough  and  Wise  (see  Appendix  C),  demonstrated  the  growing 
analytic  capabilities  of  the  DHS  s  Office  of  Risk  Management  and  Analysis  (RMA),  consisting 
of  a  set  of  interdisciplinary  modeling,  simulation,  and  risk  and  decision  analysis  approaches 
and  tools  drawn  together  over  the  past  three  years  that  support  risk  assessment.  The  presenta¬ 
tion  called  attention  to  the  potential  to  make  use  of  emergent  “in-house”  capabilities  in  regula¬ 
tory  benefits  assessment. 

Adjacent  discussions  suggested  that  the  ease  with  which  these  tools  could  be  applied  to 
specific  proposed  rules  would  depend  on  the  details  of  the  rule.  In  the  case  of  WHTI,  which 
affects  terrorism  risk  in  a  fairly  general  (i.e.,  non-scenario-specific)  way,  applying  these  tools 
might  have  taken  only  a  few  weeks  of  effort  had  the  models  been  available  at  that  time.  The 
components  of  the  models  that  address  “upstream,”  phenomena,  that  is,  activities  occurring 
prior  to  or  upon  a  terrorist  s  entry  into  the  United  States,  would  have  required  only  a  small 
number  of  changes  or  additions,  e.g.,  probability  distributions  for  modes  and  points  of  entry 
and  countervailing  risks;  the  components  of  the  models  that  address  “downstream”  phenom¬ 
ena,  that  is,  activities  occurring  after  a  terrorist  s  entry  into  the  United  States,  could  have  been 
implemented  largely  as-is. 

Exploiting  CGE  Models  to  Identify  Welfare  Effects 

CGE  models  appeared  to  present  an  attractive  but  not  fully  exploited  option  for  assessing 
the  benefits  of  terrorism  security  regulations.  A  CGE  model  can  be  used  to  identify  “wel¬ 
fare  effects,”  net  of  transfers,  including  those  stemming  from  the  indirect  and  induced  losses 
that  might  cascade  through  the  economy  in  the  event  of  a  terrorist  attack.^  Notwithstanding 
OMB  s  interest  in  separating  transfers  (2003,  p.  46),  participants  noted  that  most  economic 


^  “CGE  models  simulate  the  flow  of  commodities  and  factors  of  production  (i.e.,  labor,  capital,  and  natural  resources) 
among  producers  and  households  to  assess  how  a  change  in  policy  or  an  economic  shock  affects  the  size  and  composition  of 
the  economy”  (U.S.  Environmental  Protection  Agency,  Office  of  Air  and  Radiation,  March  2011,  p.  8-2).  In  addition,  they 
can  also  be  used  to  measure  the  net  change  in  social  welfare  (i.e.,  the  change  in  overall  consumer  and  producer  “surplus” 
across  markets)  that  results  from  shifts  in  production  or  a  reallocation  of  resources. 
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studies  of  the  consequences  of  terrorist  attacks  tend  to  focus  on  gross  “economic  impacts,”  gen¬ 
erally  measured  as  changes  in  production  or  employment.  This  is  true  even  in  those  instances 
in  which  the  studies  make  use  of  a  CGE  model.  Participants  observed  that  it  is  not  unusual  for 
agencies  to  work  with  the  results  of  economic  impact  studies  that  employ  CGE  models,  but 
that  net  changes  in  social  welfare  have  often  not  been  calculated  or  included  in  the  publication. 
Depending  on  the  study,  it  might  not  be  difficult  to  revisit  the  analysis  and  calculate  or  parse 
the  welfare  effects.  As  noted  during  the  discussion  of  Cheesebrough  and  Wise  s  presentation, 
DHS/RMA  is  adding  CGE  modeling  capacity  to  its  analytical  base  and  expects  it  to  be  fully 
operational  in  one  or  two  years.  Once  operational,  leveraging  that  internal  DHS  capability 
might  require  relatively  little  additional  effort. 

Better  Characterizing  Behavior 

Recognizing  links  between  how  individuals  and  institutions  behave  before,  during,  and  after 
terrorist  attacks,  the  underlying  risks  of  terrorist  attacks,  and  the  potential  effects  of  regulatory 
actions,  participants  discussed  a  need  to  better  characterize  behavior.  Participants  advanced 
three  general  types  of  behavior:  precautionary,  responsive,  and  adaptive.  Precautionary  behavior 
refers  to  the  actions  taken  by  individuals  and  institutions  to  mitigate  the  risk  of  attack;  respon^ 
sive  behavior  refers  to  the  actions  taken  by  individuals  and  institutions  after  an  attack  to  miti¬ 
gate  consequences  or  recover;  adaptive  behavior  refers  to  the  actions  taken  by  terrorists  to  evade 
or  work-around  security  measures.^  Participants  framed  behavioral  modeling  as  a  longer-term 
endeavor  than  leveraging  new  capabilities  or  exploiting  GGE  models. 

Discussions  about  precautionary  behavior  suggested  a  need  to  understand  how  individu¬ 
als  and  institutions  react  to  perceived  threats  and  regulatory  actions.  Individuals  and  institu¬ 
tions  may  act  on  their  own  to  reduce  the  level  of  terrorism  risk,  to  reduce  fear  or  anxiety,  or 
both.  Regardless  of  the  particular  reason,  their  behavior  may  entail  costs  to  them  and  society. 

Among  the  most  discussed  forms  of  precautionary  behavior  were  those  involving  changes 
in  individuals’  consumption  patterns;  that  is,  their  decisions  about  what  activities  to  partake  in 
and  what  purchases  to  make.  In  the  wake  of  9/11,  individuals  chose  to  fly  less  and  drive  more, 
take  fewer  vacations,  attend  New  Years  Eve  celebrations  in  Times  Square  less  heavily,  ride 
public  transit  less  often,  avoid  buying  homes  in  specific  markets,  etc.^  In  each  instance,  they 
made  consumption  decisions,  at  least  partly  motivated  by  fear,  that  left  them  less  well  off  than 
they  had  been  prior  to  9/11.  The  decisions  entailed  opportunity  costs  and,  in  some  instances, 
may  have  entailed  health  costs  to  them  and  others.  For  example,  driving  rather  than  flying 
after  the  9/11  attacks  may  have  reduced  the  fear  of  dying  in  a  terrorist  attack,  but  may  have 
resulted  in  increased  travel  time  and  perhaps  traffic  accidents  and  net  pollution  emissions.  If  a 
regulation  were  to  alleviate  individuals’  concerns  about  a  threat  and  reduce  costly  changes  in 
consumption  patterns,  then  it  could  be  said  to  have  reduced  some  of  the  expected  damages. 

Participants  posited  that  more  or  better  information  about  actual  risks,  related  to  or  apart 
from  regulatory  actions,  could  help  to  shape  perceptions  in  a  way  that  would  make  individu- 


^  These  are  not  formal  definitions,  but  reflect  the  terms  of  use,  albeit  not  uniform,  of  the  workshop  participants.  In  some 
instances,  the  participants  used  the  term  behavioral  responses  as  a  more  general  category,  encompassing  a  wide  range  of 
behaviors. 

^  The  authors  expanded  the  list  of  examples  to  include  vacations  and  housing  for  illustrative  purposes.  Participants  also 
referenced  government  decisions  to  shut  down  commerce  after  9/11  as  a  precautionary  action  leading  to  “self-inflicted 
damage.” 
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als  less  likely  to  make  costly  consumption  decisions.  But  participants  also  questioned  whether 
having  that  information  could  have  unintended  consequences;  for  example,  having  visible  evi¬ 
dence  of  risks  in  the  form  of  added  policing  might  only  serve  to  increase  individuals’  anxiety. 

The  discussion  of  precautionary  behavior  also  provided  another  opportunity  for  partici¬ 
pants  to  consider  the  difference  between  transfers  and  net  welfare  effects.  If  a  terrorism  security 
regulation  merely  reallocates  costs,  then  it  entails  “only”  a  transfer.^®  To  illustrate,  consider  a 
case  involving  commercial  property  and  fences:  Absent  a  regulation,  tenants,  fearing  intru¬ 
sions,  might  choose  to  install  fences  as  a  precautionary  measure;  if  a  security  regulation  then 
requires  that  landlords  provide  fences,  meeting  the  same  specification  and  with  no  change  in 
efficiency,  it  would  only  shift  the  cost  from  tenants  to  landlords,  who  might  still  pass  the  costs 
back  to  tenants  via  higher  rents. 

Discussions  about  responsive  behavior  were  narrower  in  scope  but  drew  attention  to  the 
possible  importance  of  distinguishing  between  long-  and  short-run  phenomena.  One  presen¬ 
tation  (see  Kousky  in  Appendix  C)  spoke  to  responsive  behavior  in  the  wake  of  Hurricane 
Katrina — e.g..  Do  people  choose  to  rebuild  or  move  out?^^ — that  might  generalize  to  other 
circumstances  requiring  similar  decisions.  In  a  later  discussion,  a  participant  noted  that  the 
relevance  of  responsive  behavior  might  be  limited  to  a  small  set  of  events,  e.g.,  an  attack  with 
biological  agents,  where  outcomes  may  depend  strongly  on  the  immediate  actions  of  individu¬ 
als  and  institutions,  but  that  in  those  instances  a  clear  understanding  of  that  behavior  could 
be  crucial. 

Discussions  about  adaptive  behavior,  like  those  of  precautionary  and  responsive  behav¬ 
ior,  also  suggested  a  need  to  understand  the  reactions  of  individuals  and  institutions,  but,  in 
this  case,  those  of  “intelligent  adversaries.”  Adaptive  behavior  came  to  light  as  an  important 
source  of  difficulty  in  anticipating  regulatory-induced  changes  in  risk.  If,  for  example,  a  ter¬ 
rorist  adapts  to  a  regulatory  action  by  altering  his  or  her  decision  about  the  location  or  mode 
of  attack,  the  action,  through  its  effect  on  the  terrorist’s  behavior,  might  also  be  altering  the 
probability  of  specific  events,  hence,  damages.  The  presentation  by  Hammitt  (see  Appendix  C), 
pointed  out  that  making  it  harder  for  a  terrorist  to  use  one  access  route  would  make  another 
route  relatively  more  attractive,  thus  creating  a  countervailing  risk.  In  the  case  of  WHTI,  a 
terrorist  arriving  in  the  United  States  might  opt  for  a  different  point  of  entry,  e.g.,  one  that  is 
not  an  official  border  crossing. 

Understanding  the  Consequences  of  Fear 

Causing  fear  in  a  population  to  achieve  some  larger  goal  is  often  a  core  motivation  of  terror¬ 
ism  (Hoffman,  2006).  To  the  extent  that  fear  manifests  itself  as  a  consequence  of  either  the 
threat  or  realization  of  a  terrorist  attack,  participants  touched  on  at  least  three  different  ways 


Though  not  addressed  explicitly  during  the  workshop,  one  could  still  argue  for  the  desirability  of  a  transfer,  e.g.,  on  the 
basis  of  distributional  considerations. 

The  authors  drew  this  example  from  the  workshop  discussions,  but  with  some  adaptation  to  simplify.  Similarly,  if  the 
government  were  to  provide  the  same  fences  with  equal  efficiency,  there  would  be  no  net  welfare  effect. 

Here,  the  discussion  also  touched  on  precautionary  behavior  insomuch  as  part  of  the  rebuilding  decision  might  involve 
choices  about  construction  technologies — e.g..  Are  buildings  being  constructed  to  withstand  stronger  winds? — that  relate 
to  individuals’  and  institutions’  experience  of  the  event  and  concerns  about  future  events. 
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that  it  might  result  in  damages. First,  as  addressed  above,  it  could  affect  decisions  people 
make  about  their  “consumption,”  which  could  entail  opportunity  and  other  costs  to  them  and 
society.  Second,  fear  resulting  from  the  threat  or  realization  of  terrorism  could  lead  to  mental 
health  trauma,  such  as  post-traumatic  stress  disorder,  anxiety,  or  depression.  Third,  short  of 
causing  changes  in  behavior  and  mental  trauma,  fear  might  reduce  a  person  s  overall  level  of 
well-being,  peace  of  mind,  or  happiness. 

Reductions  in  each  of  these  consequences  could  be  considered  among  the  benefits  of 
terrorism  security  regulations  and,  thus,  should  be  captured  in  an  assessment  of  regulatory 
benefits,  fiowever,  participants  noted  that  these  types  of  benefits,  particularly  those  relating  to 
mental  health  and  happiness,  are  rarely  incorporated  in  regulatory  assessments.  One  partici¬ 
pant  indicated  that  it  could  be  especially  difficult  to  incorporate  fear  or  anxiety  in  a  benefits 
estimate  because  they  are  not  necessarily  proportional  to  the  probability  or  likely  magnitude  of 
an  attack;  hence,  they  may  not  be  scalable  to  a  reduction  in  expected  harm.^^ 

Workshop  discussions  suggested  that  each  of  these  areas,  i.e.,  behavior,  mental  health, 
and  happiness,  presents  an  opportunity  for  basic  research  to  improve  the  science  underlying 
benefit-cost  analysis  and  its  implementation.  Changes  in  consumption  patterns  could  be  cap¬ 
tured,  in  theory,  if  the  analysis  includes  indirect  economic  effects  and  models  accurately  reflect 
behavioral  changes.  Mental  health  trauma  could  be  addressed  as  an  injury  or  illness,  but,  while 
widely  acknowledged,  it  has  not  been  well  measured  in  this  context.  The  concept  of  happiness 
is  attracting  greater  attention  in  the  area  of  welfare  economics  but,  in  its  formative  stages,  has 
not  yet  been  addressed  in  this  type  of  benefit-cost  analysis. 


Data  Collection 

Workshop  presentations  and  surrounding  discussions  acknowledged  the  scarcity  of  data  with 
which  to  assess  the  benefits  of  regulatory  action,  citing  gaps  in  collection — both  avoidable  and 
unavoidable — and  security  considerations  as  reasons  for  the  scarcity,  but  also  discussed  the 
extent  to  which  data,  per  se,  are  truly  a  limiting  factor. 

Participants  considered  expert  elicitation  and  natural  experiments  as  a  two  possible  means 
of  gathering  information  and  informing  regulatory  assessments. 

Expert  Elicitation 

Most  of  the  presentations  referenced  expert  elicitation  as  a  potentially  valid  and  valuable  source 
of  data,  fiowever,  the  discussions  during  and  after  those  presentations  highlighted  the  impor¬ 
tance  of  distinguishing  between  a  process  of  true  “expert  elicitation”  and  a  mere  gathering  of 
“expert  opinion”:  The  former  involves  empirically  validated  techniques  designed  to  reliably 
collect  unbiased  estimates,  whereas  the  latter  sometimes  involves  unstructured  queries  without 
giving  adequate  attention  to  question  framing  or  expert  selection. 


Studies  of  public  reactions  to  terrorism  have  demonstrated  how  extensive  the  reactions  of  fear  can  be  following  terrorist 
attacks  and  how  levels  of  fear  experienced  are  driven  by  combination  of  geographic  proximity,  social  connection,  and  media 
exposure  (Fischhoff  et  ah,  2003;  Silver  et  ah,  2002). 

Participants  referenced  Adler  (2004),  Sunstein  (1997  and  2003),  and  Sunstein  and  Zeckhauser  (2011)  as  sources  of 
insight  to  conceptualizing,  modeling,  and  measuring  fear. 
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In  discussing  reliable  expert  elicitation,  participants  identified  important  limitations  and 
productive  uses.  Although  the  participants  framed  expert  elicitation  as  a  plausible  means  of 
parameterizing  uncertainty,  they  emphasized  that  expert  elicitation  cannot  be  used  to  “create 
data,”  absent  underlying  knowledge,  or  be  expected  to  “eliminate  uncertainty.”  Quoting  one 
participant,  “Some  people  may  say  ‘we  don’t  have  the  data  ...  so  let’s  go  get  some  experts  to 
create  the  data,”’  but,  said  the  participant,  that  will  not  work — “if  the  experts  don’t  know,  your 
eliciting  isn’t  going  to  help.”  For  that  reason,  the  participant  suggested  the  importance  of  using 
established  criteria  for  elicitation  at  DHS,  i.e.,  that  it  be  used  only  when  it  is  possible  to  posit  an 
answerable  question  to  a  group  of  individuals  that  has  some  basis  for  making  a  judgment.  For 
example,  in  the  case  of  WHTI,  it  might  have  made  sense  to  ask  a  counterfeiting  expert  about 
the  ease  of  producing  a  passable  false  document,  but  it  might  not  have  made  sense  to  ask  him 
or  her  about  the  feasibility  of  developing  better  software  to  detect  fraudulence.^^  Reliable  expert 
elicitation  might  also  be  quite  costly  in  terms  of  time  and  funding  requirements. 

Participants  also  voiced  concerns  about  common  misconceptions  surrounding  the  inter¬ 
pretation  and  implications  of  probability  estimates.  A  group  of  experts  might,  for  example, 
conclude  that  the  risk  of  an  event — be  it  a  terrorist  attack,  a  financial  crisis,  or  an  oil  spill — is 
less  than  1  percent.  The  event  might  be  deemed  unlikely,  but  it  cannot  be  ruled  out.  If  the 
attack,  crisis,  or  spill  were  to  happen,  one  might  be  tempted  to  cite  the  event  as  evidence  of  the 
experts’  fallibility,  but  the  fact  that  it  happened  would  not  necessarily  contradict  the  experts’ 
view. 

Notwithstanding  these  limitations  and  concerns,  some  participants  argued  for  the  value 
of  expert  elicitation  in  establishing  the  bounds  of  unknowing  and,  potentially,  “closing  the 
loop”  in  break-even  analysis.  Although  calculating  a  break-even  point  does  not  require  esti¬ 
mating  the  probability  of  a  particular  event,  they  noted  that  expert  elicitation  could  be  used  to 
establish  the  likelihood  of  ending  up  above  or  below  the  break-even  point. 

The  workshop  also  addressed  the  question  of  expertise,  specifically  the  definition  of  expert 
and  the  related  challenges  of  properly  selecting  individuals  to  serve  on  expert  panels  and  appro¬ 
priately  valuing  differences  in  their  perspectives  and  judgments.  Participants  observed  that 
there  are  methods  for  dealing  with  at  least  some  of  these  issues,  such  as  “seed  questions”  for 
calibrating  results,  and  that  making  use  of  expert  elicitation  in  the  context  of  benefits  assess¬ 
ment  would  require  a  clear  understanding  of  those  methods  and  of  other  “best  practices,”  more 
generally. 

Natural  Experiments 

Participants  raised  the  possibility  of  using  natural  experiments  to  gain  insight  to  the  benefits  of 
terrorism  security  regulations.^^  Although  the  idea  emerged  from  a  discussion  about  fear,  as  the 


Example  added  by  authors  for  illustrative  purposes. 

Participants  recommended  U.S.  Environmental  Protection  Agency  (August  2011)  as  a  valuable  resource  for  investigat¬ 
ing  best  practices. 

Natural  experiments  or  quasi-natural  experiments  in  economics  are  serendipitous  situations  in  which  persons  are 
assigned  randomly  to  a  treatment  (or  multiple  treatments)  and  a  control  group,  and  outcomes  are  analyzed  for  the  purposes 
of  putting  a  hypothesis  to  a  severe  test;  they  are  also  serendipitous  situations  where  assignment  to  treatment  “approximates” 
randomized  design  or  a  well-controlled  experiment.  (DiNardo,  2008) 

For  the  purposes  of  benefit-cost  analysis,  the  regulatory  action  or  change  would  fill  the  role  of  “treatment.”  If,  for 
example,  an  airport  security  measure,  such  as  a  new  passenger  or  baggage  screening  technology,  is  implemented  as  a  pilot 
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discussion  unfolded,  it  became  apparent  that  the  approach  could  shed  light  on  a  larger  number 
of  issues  pertaining  to  regulatory  effects  and  processes.  One  participant,  noting  the  difficulty 
of  quantifying  fear,  wondered  whether  it  would  be  possible  to  roll  out  a  regulation  in  a  way  that 
would  create  a  natural  experiment  and  support  an  “event  analysis.”  Others  indicated  “yes,”  and 
pointed  to  pilot  programs,  interim  final  rules,  and  staggered  or  phased  implementation,  e.g., 
WHTI,  as  variations  on  the  theme.  They  also  noted  important  limitations;  for  example,  pilot 
testing  must  occur  in  an  environment  that  is  representative  of  the  larger  policy  environment. 
If  a  port  security  program  is  tested  in  ports  that  are  very  small  or  otherwise  differ  from  ports 
of  concern,  a  study  of  the  pilot  program  might  provide  little  insight  to  the  ultimate  effects  of 
promulgation.  The  discussions  suggested  that  such  experiments  might  provide  insight  to  other 
types  of  consequences  and  provide  valuable  information  for  both  developing  regulations  and 
fine-tuning  actions. 


Cross-Cutting  Themes 

Issues  of  analytical  capacity,  transparency,  and  presentation  also  featured  prominently  in  the 
discussions.  These  issues  relate  to  modeling  (qualitative  or  quantitative)  and  data  collection. 

Modeling  Capacity 

Discussions  throughout  the  workshop  suggested  that  an  initial  investment  in  a  strong  ana¬ 
lytical  base — that  is,  a  set  of  broadly  applicable  descriptive  and  empirically  based  models — 
could  pay  off  for  CBP  in  more  rigorous  regulatory  assessments  in  the  future.  However,  it  also 
became  apparent  during  the  presentations  that  opportunities  exist  to  make  use  of  current 
models,  including  some  models  already  in  use  in  DHS/RMA  to  support  risk  assessment.  As 
noted  above,  it  might  have  taken  relatively  little  effort  to  apply  the  DHS/RMA  models  to 
the  WHTI  assessment,  had  the  models  been  in  place  and  operational  at  that  time.  An  assess¬ 
ment  of  WHTI  would  have  required  only  a  handful  of  changes  or  additions  to  the  upstream 
(pre-entry  and  entry)  components  of  the  models  and  little  or  no  change  to  the  downstream 
(post-entry)  components.  It  might  also  be  possible  to  extract  welfare  effects  from  CGE  models, 
whether  in-house  or  external. 

Transparency  in  Analysis 

Participants  agreed  on  the  need  for  better  visibility  in  regulatory  analysis  and  pointed  to  three 
sources  of  obscurity:  concerns  about  security,  reliance  on  proprietary  models,  and  model  com¬ 
plexity.  Although  legitimate  security  concerns  and  reliance  on  proprietary  models  might  con¬ 
tinue  to  thwart  efforts  to  create  a  more  open  analytical  process,  it  seemed  plausible  that  more 
information  could  be  revealed,  with  appropriate  levels  of  care,  and  that  it  might  be  possible  to 
reduce  CBP  s  and  other  agencies’  reliance  on  proprietary  models  in  future  assessments,  per¬ 
haps  by  leveraging  some  of  the  DHS/RMA  models  cited  above.  An  agency  might  be  able  to 
show  an  analysis  when  doing  so  would  not  reveal  intelligence  sources  and  methods,  security 
vulnerabilities,  or  sensitive  defense  capabilities.  Concerns  about  complexity  pertained  not  just 
to  proprietary  models,  but  also  to  any  publically  available  or  in-house  models.  (DHS/RMA’s 


program,  it  might  be  possible  to  collect  data  on  behavior  (passenger,  equipment  operator,  etc.)  pre-  and  post-pilot  and  com¬ 
pare  that  change  with  that  of  a  control  group  that  did  not  participate  in  the  pilot  program. 
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risk  assessment  models  require  tens  of  thousands  of  inputs.)  Participants  argued  for  keeping 
things  “simple,”  whenever  simplicity  will  suffice,  in  future  benefit-cost  models.  To  the  extent 
that  the  use  of  sensitive  data,  proprietary  models,  and  complex  models  is  unavoidable,  one 
might  still  illuminate  the  process  with  more  or  better  use  of  model  verification,  validation,  and 
accreditation. 

Presentation  of  Information  and  Results 

Discussions  indicated  a  commonly  held  view  that  single  approaches,  single  numbers,  and  single 
answers  would  not  suffice  and  could  even  mislead.  One  participant  noted  that  “looking  at  the 
same  information  in  different  ways  can  help  someone  understand  it  better.”  Another  spoke  to  the 
value  of  representing  the  results  of  benefit-cost  and  break-even  analysis  as  complementary:  For 
example,  a  positive  net  result  in  a  benefit-cost  analysis  might  reflect  an  underlying  assessment 
that  the  probability  of  a  particular  type  of  attack  is  “X”;  however,  a  break-even  analysis  might 
indicate  that  the  probability  could  be  as  low  as  “Y”  and  the  benefits  would  still  exceed  the 
costs.  The  comparison  yields  a  range  that  the  decisionmaker  can  wrestle  with  and  say  “I  feel 
comfortable  it  s  [the  probability  is]  within  that  range.  .  .  .” 


SECTION  3 


Points  of  Contention 


Insomuch  as  the  workshop  yielded  any  points  of  contention,  substantial  differences  were 
most  apparent  in  participants’  views  on  the  sufficiency  of  economic  tools,  particularly  those 
geared  toward  estimating  effects,  and  the  merits  of  using  alternative  decision  criteria  to  address 
uncertainty. 

Whereas  some  participants  argued  that  available  tools  for  estimating  the  consequences  of 
terrorist  events,  including  existing  estimates  of  willingness  to  pay  for  a  value  of  statistical  life,^ 
were  good  enough  for  capturing  the  big  effects,  others  suggested  that  there  were  important 
“big  gaps,”  especially  with  regard  to  measuring  welfare  effects  and  characterizing  behavior — 
precautionary,  adaptive,  and  responsive  behavior.  The  disagreement  was  not  as  much  about  the 
facts  of  the  matter — no  one  disputed  the  existence  of  gaps — but  about  their  importance,  i.e., 
whether  the  tools  were  already  “good  enough.” 

Participants  also  disagreed  as  to  whether  a  departure  from  the  usual  benefit-cost  analy¬ 
sis  criteria,  i.e.,  efficiency  and  cost-effectiveness,  could  lead  to  advances  in  regulatory  analysis. 
Some  participants  suggested  working  with  alternative  decision  criteria,  such  as  “robustness,”  to 
address  the  challenges  of  extreme  uncertainty;  however,  another  pushed  back,  describing  the 
use  of  alternative  criteria  as  potentially  “dangerous  because  it  can  be  misunderstood  as  hiding 
assumptions.”  For  example,  if  evaluating  a  terrorism  security  regulation  for  robustness,  one 
might  test  whether  it  works  well  over  a  large  share  of  scenarios,  such  as  terrorist  attacks  that 
use  different  ports  of  entry  or  weapons.^  However,  in  doing  so,  one  would  need  to  assume,  at 
least  implicitly,  that  the  selected  scenarios  were,  in  fact,  the  likely  scenarios.  On  that  basis, 
one  might  find  themselves  assuming  away  a  core  problem,  or  even  the  core  problem,  faced  by 
benefit-cost  analysis:  a  fundamental  lack  of  knowledge  as  to  which  scenarios  are  likely.  Alter¬ 
native  decision  criteria  might  be  seen  as  skirting  uncertainties  around  “changes  in  harms”  or 
“probability.”  The  participant  urged  explicit  consideration  of  the  magnitudes  of  both:  “looking 
at  just  the  probability  is  no  good,”  “looking  at  just  the  consequences  is  no  good,”  and  “any¬ 
thing  that  tries  to  pretend  you  cannot  look  at  the  whole  picture  can  be  misleading.” 


^  The  “value  of  statistical  life”  refers  to  the  measurement — monetized  value — of  society’s  willingness  to  pay  for  a  marginal 
reduction  in  the  risk  of  premature  death  (OMB,  2003,  p.  29;  Viscusi  and  Aldy,  2003).  Participants  engaged  in  little  discus¬ 
sion  about  the  value  of  statistical  life,  but  indicated  general  support  for  the  approach  and  some  interest  in  further  exploring 
whether  the  value  of  statistical  life  might  be  different — specifically,  higher — in  the  context  of  terrorism  security  regulations 
than  in  other  policy  contexts  because  of  catastrophe  aversion  and  dread.  They  were  not  certain  whether  the  literature,  to 
which  some  in  the  room  had  contributed,  could  weigh  in  definitively  and  were  skeptical  as  to  whether  the  difference  would 
matter  analytically.  (For  additional  information,  see,  e.g.,  Robinson,  2008,  Robinson  et  ah,  2010,  and  Viscusi,  2009.) 

^  Examples  added  by  authors  for  illustrative  purposes. 
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SECTION  4 


Recommendations 


The  foregoing  distillation  and  synthesis  of  emerging  ideas,  recurring  themes,  and  points  of  con¬ 
tention  suggests  several  recommendations  to  assist  CBP  in  meeting  the  challenges  of  improv¬ 
ing  the  benefit-cost  analysis  of  terrorism  security  regulations.  Whether  taken  individually  or  in 
combination,  none  of  these  recommendations  can  address  all  of  CBP  s — and  other  such  agen¬ 
cies’ — methodological  needs,  but  they  can  help  to  provide  a  stronger  foundation  for  benefit- 
cost  analysis  in  the  future. 

•  Aim  for  break-even  work  that  is  more  descriptive.  Comprehensive  or  “full  ”  benefit- 
cost  analysis  is  likely  neither  attainable  nor  desirable,  given  the  extent  of  uncertainty  and 
related  analytical  challenges,  but  break-even  analysis  can  be  used  to  explore  the  implica¬ 
tions  of  uncertainty  and  describe  conditions  under  which  benefits  could  exceed  costs  for 
particular  regulations. 

•  Incorporate  “storytelling”  into  regulatory  analysis  and  consider  adopting  logic 
modeling  as  part  of  the  regulatory  development  process.  Logic  models  can  be  used 
to  support  benefit-cost  analysis,  break-even  analysis,  and  regulatory  development,  more 
generally,  by  helping  to  identify  and  articulate  plausible  links  between  regulatory  actions 
and  outcomes. 

•  Strengthen  internal  modeling  capacity  and  leverage  existing  risk  assessment  meth¬ 
ods,  when  possible,  including  DHS/RMA  and  CGE  models.  A  varied,  multidisci¬ 
plinary  toolkit  can  be  used  to  assess  big  effects  and  shed  light  on  uncertainty.  For  exam¬ 
ple,  CGE  models  can  be  used  to  calculate  welfare  effects;  DHS/RMA’s  risk  assessment 
models  to  estimate  risk;  and,  as  already  noted,  logic  models  to  “tell  the  story”  of  a  regula¬ 
tory  action  and  its  effects. 

•  Develop  and  adopt  DHS-wide  standards  for  reliable  expert  elicitation.  Learning  more 
about  best  practices — to  better  understand  the  technique’s  strengths,  weaknesses,  and 
applicability  to  benefit-cost  analysis — and  establishing  criteria  based  on  those  practices 
would  help  to  ensure  that  expert  elicitation  throughout  DHS  produces  reliable  results. 

•  Improve  basic  science  in  potentially  important  but  under-explored  areas  of  terror¬ 
ism  consequences,  including  behavioral  responses,  mental  health,  and  fear.  Not 
only  does  fear  have  behavioral  implications  that  can  impact  the  probability  or  magnitude 
of  effects,  it  may  have  direct  bearing  on  an  individual’s  mental  health,  well-being,  and 
happiness. 

•  Incorporate  pilot  studies  and  other  natural  experiments  into  regulatory  design. 

Regulatory  rollouts  that  create  natural  experiments,  e.g.,  through  pilot  programs,  interim 
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final  rules,  or  phased  implementation,  may  yield  data  and  enable  analysis  that  provides 
insight  to  the  effects  of  final  rules. 

•  Improve  transparency.  It  may  be  possible  to  improve  the  transparency  of  benefit-cost 
analysis,  e.g.,  by  sharing  information  when  doing  so  would  not  jeopardize  security  and 
expanding  verification  and  validation. 
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APPENDIX  B 

Workshop  Agenda 


8:00-8:30  am 

Registration  and  Continental  Breakfast 

8:30-8:45  am 

Introduction  and  Format  for  Workshop 

Moderator:  Tom  LaTourrette  (RAND) 

8:45-9:30  am 

Overview  of  Challenges  Identified  During  the 

Development  of  Estimates  of  the  Benefits  of 

U.S.  Customs  and  Border  Protection  Regulations 

Presenter:  Jennifer  Baxter  (lEc) 

9:30-10:30  am 

Applying  Modeling  and  Simulation  to  Estimate 

Risk  Reduction  Benefits  for  Regulatory  Benefit-Cost 

Analysis 

Presenter:  Tony  Cheesebrough  (DHS,  RMA) 

10:30-10:45  am 

Coffee  Break 

10:45-11:45  am 

Using  Logic  Models  to  Assess  Security  Benefits 

Presenter:  Victoria  A.  Greenfield  (RAND) 

11:45-12:45  pm 

Characterizing  Benefits  of  Anti-Terrorism  Rules 

Presenter:  James  K.  Hammitt 

(Harvard  University  and  Toulouse  School  of 

Economics) 

12:45—1:30  pm 

1:30-2:30  pm 

Lunch 

Improving  Estimation  of  the  Benefits  of  Terrorism  Risk 
Reduction:  Learning  from  Environmental  Economists? 
Presenter:  Carolyn  Kousky  (Resources  for  the  Future) 

2:30-3:30  pm 

Summary  and  Discussion  of  Observations 

Moderator:  Henry  H.  Willis  (RAND) 

3:30-3:45  pm 
3:45-5:00  pm 

Coffee  Break 

Summary  and  Discussion  of  Observations  (cont.) 

Moderator:  Henry  H.  Willis  (RAND) 
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APPENDIX  C 


Invited  Presentations  from  Workshop  Participants 


This  appendix  contains  five  presentations:  the  four  invited  presentations  and  an  overview  (see 
below).  The  presentations  are  organized  in  the  order  in  which  they  were  given  during  the  work¬ 
shop.  They  have  not  undergone  peer  review  or  formal  editing.  By  and  large,  they  appear  here 
much  as  they  appeared  during  the  workshop,  with  only  minor  clarifications,  typographical 
corrections,  and  formatting  changes. 

•  Jennifer  Baxter,  “Overview  of  Challenges  Identified  During  the  Development  of 
Estimates  of  the  Benefits  of  U.S.  Customs  and  Border  Protection  Regulations,”  lEc. 

•  Tony  Cheesebrough  and  Ryan  Wise,  DHS/RMA,  “Applying  Modeling  and  Simulation 
to  Estimate  Risk  Reduction  Benefits  for  Regulatory  Benefit-Cost  Analysis,”  DHS/ 

RMA. 

•  Victoria  A.  Greenfield,  Henry  H.  Willis,  and  Tom  LaTourrette,  “Using  Logic  Models  to 
Assess  Security  Benefits,”  RAND. 

•  James,  K.  Hammitt,  “Characterizing  Benefits  of  Anti-Terrorism  Rules,”  Harvard 
University  (Center  for  Risk  Analysis)  and  Toulouse  School  of  Economics 
(LERNA-INRA). 

•  Carolyn  Kousky,  “Improving  Estimation  of  the  Benefits  of  Terrorism  Risk  Reduction: 
Learning  from  Environmental  Economists?”  Resources  for  the  Future. 
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Overview  of  Challenges  Identified  During  the  Development  of  Estimates  of 
the  Benefits  of  U.S.  Customs  and  Border  Protection  Regulations 

Presenter:  Jennifer  Baxter  (lEc) 


lEc 


OVERVIEW  OF  CHALLENGES  IDENTIFIED 
DURING  THE  DEVELOPMENT  OF  ESTIAAATES 
OF  THE  BENEFITS  OF  U.S.  CUSTOMS  AND 
BORDER  PROTECTION  REGULATIONS 

Prepared  by: 
Jennifer  Baxter 


November  29,  201 1 
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Background 


•  Congress  passes  laws  governing  the  United  States  and  authorizes  U.S.  Customs  and  Border 
Protection  (CBP)  to  put  those  laws  into  effect  by  promulgating  regulations. 

•  The  amount  of  discretion  available  to  CBP  as  it  crafts  regulations  depends  on  the  language 
of  the  specific  law.  For  example: 

■  Intelligence  Reform  and  Terrorism  Prevention  Act  of  2004  (IRTPA):  Requires  that  the 
Secretary  of  Homeland  Security,  in  consultation  with  the  Secretary  of  State,  develop  and 
implement  a  plan  to  require  travelers  entering  the  United  States  to  present  a  passport, 
other  document,  or  combination  of  documents,  that  are  “deemed  by  the  Secretary  of 
Homeland  Security  to  be  sufficient  to  denote  identity  and  citizenship.”  (Section  7209) 

■  Maritime  Transportation  Security  Act  of  2002  (MTSA):  Requires  that  the  Secretary  of 
Homeland  Security,  in  consultation  with  the  Transportation  Security  Oversight  Board, 
develop  a  program  to  evaluate  and  certify  secure  systems  of  international  intermodal 
transportation,  including  “...developing  performance  standards  to  enhance  the  physical 
security  of  shipping  containers,  including  standards  for  seals  and  locks”  (Section 
70116(b)(3)) 

■  Security  and  Accountability  for  Every  Port  Act  of  2006  (SAFE  Port  Act):  The  Secretary  of 
Homeland  Security  “shall  require  the  electronic  transmission  to  the  Department  of 
additional  data  elements  for  improved  high-risk  targeting,  including  appropriate  elements 
of  entry  data...to  be  provided  as  advanced  information  with  respect  to  cargo  destined  for 
importation  into  the  United  States  prior  to  loading  of  such  cargo  on  vessels  at  foreign 
ports.”  (Section  203(b)). 
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Why  does  CBP  undertake  benefit-cost  analysis? 


•  In  certain  circumstances.  Congress  explicitly  directs  CBP  to  consider  economic  information  as  it 
promulgates  regulations.  For  example: 

■  In  promulgating  the  Importer  Security  Filing  and  Additional  Carrier  Requirements  (10+2)  rule,  the  SAFE  Port 
Act  required  CBP  to  “consider  the  cost,  benefit,  and  feasibility”  of  the  proposed  action.  (Section 
203(c)(1)). 

■  For  the  recent  visa  waiver  program  for  Guam  and  the  Commonwealth  of  the  Northern  Mariana  Islands 
(CNMI),  Congress  required  a  “listing  of  any  Country  from  which  the  Commonwealth  has  received  a 
significant  economic  benefit  from  the  number  of  visitors  for  pleasure  within  the  one-year  period  preceding 
the  date  of  enactment...unless  the  Secretary  of  Homeland  Security  determines  that  such  country’ s 
inclusion  on  such  list  would  represent  a  threat  to  the  welfare,  safety,  or  security  of  the  United  States  or  its 
territories...  (Consolidated  Natural  Resources  Act  Section  702(b)) 

•  Absent  explicit  direction  from  Congress,  several  other  laws  and  executive  orders  compel  consideration  of 
economic  information. 


■  Executive  Order  12866  Regulatory  Planning  and  Review  (1993)  directs  federal  agencies  to  “assess  the 
costs  and  benefits  of  the  intended  regulation  and,  recognizing  that  some  costs  and  benefits  are  difficult  to 
quantify,  propose  or  adopt  a  regulation  only  upon  a  reasoned  determination  that  the  benefits  of  the 
intended  regulation  justify  its  costs.”  (Section  1(b)(6)) 

■  Executive  Order  1 3563  Improving  Regulation  and  Regulatory  Review  (201 1 )  reaffirms  the  direction  and 
principals  provided  in  Executive  order  12866. 

■  The  Regulatory  Flexibility  Act  of  1980  (RFA),  as  amended  by  the  Small  Business  Regulatory 
Enforcement  Fairness  Act  of  1996  (SBREFA),  requires  federal  agencies  to  consider  the  economic  impacts 
of  their  regulations  on  small  entities  to  minimize  those  impacts. 

■  The  Unfunded  Mandates  Reform  Act  of  1995  (UMRA)  requires  federal  agencies  to  assess  the  economic 
effects  of  their  regulations  on  state,  local,  and  tribal  governments  and  the  private  sector. 
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How  is  benefit-cost  analysis  used? 


•  A  well-crafted  benefit-cost  analysis  following  best  practices  laid  out  in  0MB’ s 
Circular  A-4  contributes  to  the  regulatory  development  process  in  several  ways: 

■  Consideration  of  regulatory  alternatives:  0MB  directs  agencies  to  consider  the 
benefits  and  costs  of  the  proposed  regulation  and  several  alternatives.  Depending  on 
data  availability,  the  economic  analysis  can  identify  efficient  alternatives  (net 
benefits  are  positive)  and  information  about  the  cost-effectiveness  of  each  option. 

■  Baseline  definition:  Circular  A-4  directs  Agencies  to  clearly  define  the  world  without 
the  regulation.  This  process  and  the  collection  of  necessary  data  often  highlight 
important  attributes  of  affected  environment  that  influence  the  development  of  the 
rule. 

■  Predicting  responses  to  the  regulation:  Economic  information  about  preferences 
can  highlight  unanticipated  positive  and  negative  effects  of  the  proposed  regulation. 

■  0MB  clearance:  If  the  economic  analysis  is  insufficient,  0MB  may  return  the  rule  to 
CBP  for  further  consideration,  delaying  or  a  proposed  rule  or  forcing  the  Agency  to 
start  over. 

■  Protection  against  future  litigation:  While  citizen  suits  are  not  permitted  under 
Executive  Orders,  Agencies  can  be  sued  for  failing  to  comply  with  the  RFA  or  UMRA. 
Furthermore,  to  the  extent  that  the  law  necessitating  the  regulation  requires 
economic  analysis,  citizen  suits  over  the  Agency’ s  consideration  of  the  best  available 
economic  information  are  possible. 
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Defining  benefits 


•  OMB  directs  Agencies  to  measure  regulatory  benefits  in  welfare 
terms,  i.e.,  estimates  of  how  much  individuals  are  willing  to  pay  for 
the  proposed  improvement 

■  Generally,  it  is  difficult  to  obtain  a  single,  holistic  estimate  of 
willingness  to  pay  for  a  program  or  regulation 

■  Economists  often  apply  a  damage  function  approach 

•  In  the  context  of  CBP’ s  security  regulations,  application  of  either 
approach  requires  information  about  the  baseline  security  risk  and 
incremental  changes  in  risk  resulting  from  proposed  regulation 

•  CBP  lacks  information  quantifying  the  risk  change  resulting  from  its 
regulatory  alternatives.  In  addition,  if  information  about  such  risk 
changes  were  available,  data  on  the  value  of  these  changes  is 
subject  to  considerable  uncertainty.  As  a  result,  benefit-cost 
analysis  cannot  play  its  traditional  role  in  the  development  of  CBP 
regulations. 
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Necessary  risk  information  is  limited 


•  Early  attempts  by  CBP  to  estimate  the  risk  reductions  resulting  from  proposed 
regulations  encountered  several  challenges: 

■  Unlike  natural  disasters,  the  historical  record  on  the  frequency  and  severity  of 
terrorist  events  in  the  United  States  is  too  small  to  provide  a  meaningful  basis  for 
projecting  future  risks. 

■  The  development  of  new  risk  models  was  difficult  to  implement  in  the  typical 
timeframe  for  promulgating  a  regulation. 

-  Baseline  risk  modeling  efforts  in  other  parts  of  DHS  required  the  collection  of  a 
significant  amount  of  new  data  and  relied  on  expert-elicitation  processes  to  estimate 
the  probability  of  different  threat-consequence  scenarios. 

-  Evaluation  of  the  efficacy  of  different  regulatory  alternatives  required  new  rounds  of 
expert  elicitation. 

-  The  models  often  relied  on  classified  information  that  could  not  be  included  in  the 
publicly-available  administrative  record  supporting  new  regulations. 

•  Furthermore,  interest  in  investing  the  time  and  effort  in  detailed  benefit-cost 
analysis  was  limited  as  there  was  a  general  sense  that  any  rule  potentially 
preventing  another  9/11  attack  as  worth  it,”  regardless  of  the  regulatory 
costs. 
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The  near-term  solution...break-even  analysis 


•  In  2005,  faced  with  increased  scrutiny  from  0MB  regarding  the  potential  costs 
and  benefits  of  proposed  security  regulations,  CBP  began  using  break-even 
analysis. 

•  According  to  0MB, 

It  will  not  always  be  possible  to  express  in  monetary  units  all  of  the  important  benefits 
and  costs. ...If  the  non-quantified  benefits  and  costs  are  likely  to  be  important,  you 
should  carry  out  a  “threshold”  analysis  to  evaluate  their  significance.  Threshold  or 
“break-even”  analysis  answers  the  question,  “How  small  could  the  value  of  the  non- 
quantified  benefits  be  (or  how  large  would  the  value  of  the  non-quantified  costs  need  to 
be)  before  the  rule  would  yield  zero  net  benefits?”  (Circular  A-4,  p.  2) 

•  CBP  has  implemented  two  different  approaches  to  break-even  analysis  of 
security  regulations: 

■A  scenario-based  approach  that  estimates  the  necessary  reduction  in  the  probability  of 
a  specific  type  of  terrorist  attack;  and 

■A  broad-based  approach  that  relies  on  estimates  of  total  baseline  terrorism  risk  in  the 
U.S.  and  estimates  the  necessary  percentage  reduction  in  that  baseline  risk 
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Limitations  of  scenario- based  break-even  analysis 


•  Scenarios  selected  for  connparison  must  reflect  the  types  of  events  likely  to  be  affected  by  the 
regulation 

■  Requires  understanding  of  plausible  linkages  between  the  proposed  action  (e.g.,  requiring  individuals  to 
show  a  valid  passport  to  enter  the  United  States)  and  the  types  of  events  likely  to  be  attempted  absent  the 
regulation  (e.g.,  detonation  of  a  bomb  on  a  train  or  airplane). 

■  For  regulations  aimed  at  keeping  harmful  people  or  weapons  out  of  the  U.S.,  it  may  be  difficult  to  capture 
the  full  range  of  plausible  events. 

•  Requires  the  valuation  of  the  “avoided  costs”  of  terrorist  events.  Data  necessary  to  monetize  these 
costs  may  be  limited  or  require  integration  into  more  sophisticated  models  to  estimate  welfare  losses. 

■  Much  of  the  economic  research  published  to  date  on  terrorist  events  focuses  on  the  regional  economic 
impacts  of  such  events,  rather  than  the  change  in  welfare  experienced  by  the  U.S.  as  a  whole.  Additional 
analysis  of  market  data  in  these  studies  to  estimate  welfare  losses  is  possible  but  often  requires 
coordination  with  the  original  authors  of  the  studies  and  additional  information  about  behavioral  response 
to  market  changes.  Where  such  effort  is  not  feasible,  we  often  extract  from  these  studies  only  the 
elements  of  the  damage  estimates  that  we  can  be  certain  represent  losses  to  society  (rather  than 
transfers),  such  as  casualty  counts  and  property  damage.  As  a  result,  estimates  of  avoided  costs  may  omit 
key  cost  categories  (e.g.,  the  opportunity  cost  of  business  interruption). 

■  Methods  for  valuing  avoided  fatalities  and  nonfatal  injuries  exist;  however,  DHS  decision-makers  question 
whether  benefits  transfer  of  willingness  to  pay  to  avoid  fatalities  associated  with  more  familiar  risks  (e.g., 
workplace  fatalities)  understates  the  benefits  of  security  regulations.  They  generally  believe  that 
individual  willingness  to  pay  to  reduce  terrorism-caused  fatalities  or  injuries  is  likely  to  be  larger  than  the 
values  for  other  types  of  risk  reductions;  however,  the  existing  literature  does  not  necessarily  confirm  this 
belief. 

■  Even  if  total  avoided  costs  can  be  estimated,  such  an  ex-poste  approach  to  valuation  may  understate 
willingness  to  pay  to  avoid  the  event. 
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Limitations  of  scenario-based  break-even  analysis 
(continued) 


•  The  results  of  the  break-even  analysis  are  difficult  to  interpret. 

■  The  baseline  probability  of  the  subject  attack  is  unknown.  As  a  result,  it  can  be 
difficult  to  evaluate  whether  the  critical  change  in  risk,  e.g.,  the  need  to  avoid  one 
event  every  10  years,  is  feasible  given  the  existing  threat. 

•  The  approach  focuses  on  reductions  in  the  number  of  successful  attacks,  rather 
than  the  potential  for  the  proposed  rule  to  reduce  the  severity  of  the 
consequences  of  an  attack.  For  example,  a  proposed  rule  limiting  the  number 
of  terrorist  who  are  able  to  enter  the  United  States  may  not  only  reduce  the 
likelihood  of  an  event;  it  may  also  force  terrorist  groups  to  focus  on  smaller- 
scale  events  requiring  less  manpower. 

•  The  simplicity  of  the  model  does  not  allow  analysts  to  evaluate  reductions  in 
the  probability  of  multiple  types  of  attacks  simultaneously.  Given  the  generally 
broad  nature  of  CBP’ s  regulations,  it  is  unlikely  that  a  proposed  regulation 
affects  a  only  one  attack/consequence  scenario. 

•  This  simplistic  approach  does  not  account  for  target-shifting.  That  is,  although 
the  risk  of  one  type  of  event  may  decrease,  terrorists  may  shift  focus  to  another 
type  of  event,  resulting  in  an  offsetting  increase  in  risk. 
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Limitations  of  break-even  analysis  relying  on  broad 
estimates  of  baseline  terrorism  risk 


•  This  approach  is  more  appealing  because  it  addresses  several  key  failings  of  the  previous, 
scenario-based  approach. 

■  Assuming  the  model  of  terrorism  risk  is  robust,  regulators  have  a  baseline  against  which 
they  can  evaluate  the  feasibility  of  the  required  risk  reduction. 

■  It  implicitly  allows  for  the  consideration  of  multiple  attack/consequence  scenarios 
simultaneously. 

■  By  focusing  on  the  break-even  “risk”  reduction  as  opposed  to  changes  in  the  probability  of 
events,  it  captures  potential  changes  in  both  probability  and  consequences. 

•  However,  it  is  also  subject  to  several  limitations,  including: 

■  The  models  are  expensive  to  build  and  maintain,  and  often  contain  proprietary  or  classified 
information.  As  a  result,  they  may  not  be  appropriate  for  a  rulemaking  process  that 
requires  transparency  and  public  disclosure  of  information. 

■  Existing  models  may  not  include  a  complete  accounting  of  terrorism  risk.  For  example,  the 
model  used  in  the  Western  Hemisphere  Travel  Initiative  (WHTI)  analysis  was  developed  for 
the  insurance  industry,  and  therefore  only  includes  people  or  assets  that  are  insurable. 

The  model  omits  the  value  of  at-risk  resources  not  subject  to  insurance  (e.g.,  government 
property,  uninsured  individuals). 

■  The  models  may  value  the  consequences  of  terrorist  attacks  in  ways  that  are  inconsistent 
with  welfare  economic  theory  or  current  best  practices  applied  by  CBP.  It  may  be  difficult 
to  update  values  assumptions  for  use  in  regulatory  analysis. 
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Concluding  thoughts  about  break-even  analysis 


•  Both  break-even  approaches  previously  employed  by  CBP  suffer  from 
two  basic  flaws: 

■  They  cannot  provide  information  about  whether  the  proposed  rule  is 
economically  efficient. 

■  They  do  not  allow  for  meaningful  comparison  of  regulatory  alternatives 
where  the  costs  of  the  alternatives  are  similar. 

•  Break-even  analysis  is  most  useful  in  two  situations: 

■  Highlighting  extreme  cases,  i.e.,  regulations  where  the  costs  are  so  large 
relative  to  the  resources  at  risk  that  net  positive  benefits  are  unlikely. 

■  Providing  additional  information  related  to  the  size  of  the  “gap”  between 
quantified  costs  and  non-risk  related  benefits,  such  as  quantifiable  time 
savings. 

•  Application  of  the  scenario- based  approach  may  have  some  use  in 
addressing  0MB’ s  direction  to  describe  the  link  between  the  proposed 
action  and  desired  outcome,  as  this  information  is  necessary  to  justify 
the  selection  of  consequence  scenarios. 
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Ancillary  Benefits 


For  rules  where  analysts  are  unable  to  monetize  or  even  quantify 
intended  benefits,  Agencies  may  have  incentive  to  work  harder  to 
identify,  quantify,  and/or  monetize  ancillary  benefits. 

0MB  states  that  consideration  of  ancillary  benefits  is  appropriate. 

■  Analysts  should  “Identify  the  expected  undesirable  side  effects  and 
ancillary  benefits  of  the  proposed  regulatory  action  and  the 
alternatives.  These  should  be  added  to  the  direct  benefits  and 
costs  as  appropriate.”  {Circular  A-4,  p.  3) 

Monetization  of  ancillary  benefits  could  be  incorporated  into  break¬ 
even  analysis,  reducing  the  required  risk  reductions. 

Decision-makers  should  carefully  consider  the  appropriateness  of 
justifying  a  proposed  rule  based  solely  on  ancillary  benefits, 
particularly  where  quantification  of  these  benefits  is  subject  to 
significant  uncertainty  and/or  these  benefits  deviate  substantially 
from  the  purpose  and  intent  of  the  subject  law  or  the  Agency’ s 
mission. 
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RMA  Overview 


■  Founded  in  April  2007  as  part  of  the 
National  Protection  and  Programs 
Directorate  to  enable  and  advance  the 
effective  management  of  risk  by  the 
homeland  security  enterprise 

■  39  Full  Time  Equivalents  (FTEs) 

■  Variety  of  skill  sets 

■  Economics  and  operations  research 

■  Quantitative  policy  analysis 

■  Engineering  and  physical  sciences 

■  Business  and  public  administration 

■  Computer  science  and  information  systems 

■  Located  near  Gallery  Place/  Chinatown 


“We  must  apply  a  risk-based  framework  across  all  homeland  security  efforts  in  order  to  identify 
and  assess  potential  hazards... determine  what  levels  of  relative  risk  are  acceptable,  and  prioritize 
and  allocate  resources  among  all  homeland  security  partners.”  (National  Homeland  Security  Strategy) 


Homeland 

Security 


TRUSTED  ANALYSIS, 

BETTER  DECISIONS, 

SAFER  COMMUNITIES.  3 


Invited  Presentations  from  Workshop  Participants  35 


Quantitative  definition  of  risk 

■  In  analyzing  risk  we  are  attempting  to  envision 
how  the  future  will  turn  out  if  we  undertake  a 
certain  course  of  action  (or  inaction). 

■  Fundamentally,  risk  analysis  consists  of  an 
answer  to  the  following  three  questions  (Kaplan 
and  Garrick,  1981): 

■  What  can  happen?  (i.e..  What  can  go 
wrong?) 

■  How  likely  is  it  to  happen? 

■  If  it  does  happen,  what  are  the 
consequences? 
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Modeling  and  simulation  to  estimate  risk 


■  Models  are  simplified  representations  of  reality  and  can  take  many  forms: 
physical,  conceptual,  mathematical,  etc. 

■  More  formally,  models  are  approximations,  representations,  or  idealizations 
of  selected  aspects  of  the  structure,  behavior,  operation,  or  other 
characteristics  of  a  real-world  process,  concept,  or  system. 

■  In  simplified  terms,  we  model  risk  as  sets  of  triplets  that  answer  the  three  risk 
questions: 

q> 

■  where  is  an  event  identification  or  description; 

■  Pi  is  the  probability  of  that  event;  and 

■  Ci  is  the  consequence  or  evaluation  measure  of  that  event,  i.e.,  the  measure  of 
damage. 

■  Simulations  are  models  that  behave  or  operate  like  a  given  process,  concept,  or 
system  when  provided  a  set  of  controlled  inputs. 

■  In  technical  terms,  we  implement  our  risk  models  as  stochastic,  discrete  event 
simulations  using  Monte  Carlo  computational  algorithms  that  rely  on 
repeated  random  sampling  to  compute  results. 
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Visualizing  model  structure  as  an  event  tree 
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Example:  baseline  event  risk  prior  to  WHTI 
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Example:  baseline  event  risk  prior  to  WHTI,  (continued) 
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Example:  event  risk  following  WHTI  implementation 
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Functional  form  parameters 

Risk  per  event  =  Frequency  x  Branch  Probability  x  Consequences 
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Risk  reduction  functional  form  for  single  program  and  event 

■  Reduced  risk  is  the  difference  in  risk  associated  with  programs. 

Risk  reduction  for  a  given  program  k  for  any  event  e  within  an  incident  set  J  is  the  product  of  event  e’s 
probability  and  consequences,  when  all  programs  (except  k)  are  accounted  for. 

Human  risk  reduction  per  year  for  program  k  acting  on  event  e  within  incident  set  i 
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Model  inputs 


What  can  go  wrong? 

■  Scenarios  identify  threats  of  concern  by  attack  method,  weapon  type,  and  target  classes  for  specific 
types  of  nuclear,  radiological,  biological,  chemical,  and  explosive  terrorist  attacks. 

■  Incident  chains  with  mutually  exclusive  options  are  also  identified  (e.g.,  mode  and  point  of  entry). 

If  it  does  happen,  what  are  the  consequences? 

■  Fatalities  are  estimated  using  models  such  as  Hazard  Prediction  and  Assessment  Capability 
(HPAC),  a  physics-based  DOD  model  that  estimates  deaths  based  on  exposure  to  toxic  material 
dispersal  in  terms  of  blast  radii  (for  nuclear)  or  plume  size  (in  the  case  of  CRN). 

■  Economic  consequences  are  modeled  in  terms  of: 

Direct  impacts,  which  are  changes  in  the  output  of  industries  immediately  affected  by  the  attack, 
including  industries  that  provide  remediation  activities  and  industries  that  experience  reduced  business 
volume  due  to  public  avoidance  of  the  target  area; 

■  Indirect  impacts,  or  changes  in  output  of  industries  that  sell  goods  and  services  to  an  industry  directly 
affected  by  the  attack;  and 

■  Induced  impacts,  or  changes  in  output  of  all  industries  caused  by  changes  in  household  income. 

How  likely  is  it  to  happen? 

Historical  data  on  terrorism  is  sparse  (fortunately);  therefore,  subject  matter  experts  in  the 
intelligence  community  are  elicited  for  judgments  on  scenario  and  incident  chain  probabilities. 

■  Programmatic  SMEs  are  elicited  for  judgments  on  the  effectiveness  of  countermeasures. 
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Expert  judgment  in  risk  analysis 


■  Judgments  are  necessarily  used  in  the  study  of  any  complex  technical  problem, 
and  throughout  any  analysis,  expert  judgment  is  essential. 

■  For  example,  judgments  are  made  in  the  selection  of  models  and  analysis  forms. 

■  Data  sources  need  to  be  identified  and  their  use  has  to  be  assessed.  If  judged  useful,  data 
have  to  be  collected  and  interpreted. 

■  While  expert  judgment  is  always  used,  this  use  has  often  been  informal,  implicit 
and  undocumented. 

■  Verbal  or  qualitative  judgments  lack  a  common  basis  for  interpretation  and 
comparison,  while  making  judgments  quantitative  allows  them  to  be  combined 
with  other  sources  of  information  and  to  be  manipulated  in  models. 

■  Quantification  does  not  mean  certainty;  rather,  quantification  allows  experts  to 
more  clearly  indicate  how  uncertain  they  are  about  an  issue  or  topic,  making  it 
harder  for  others  to  misconstrue  or  misuse  results  of  an  assessment. 

■  Capturing  uncertainty  is  critical,  and  processes  have  been  designed  to  do  so  in  a 
consistent  and  appropriate  manner,  eliciting  distributions  or  ranges,  not  only  averages. 

■  Central  values  or  point  estimates  are  never  reported  on  their  own  -  uncertainty  is  always 


characterized. 
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History  and  current  use  of  expert  elicitation  for  rare  events 


■  WASH-1400  -  1975:  one  of  the 

first  demonstration  of  probabilistic 
risk  assessment  as  a  method  for 
tackling  the  probability  estimation 
problem  for  low-frequency  events. 

Assessed  accident  risk  for  nuclear 
power  plants  (probability  of 
complete  core  meltdown  assessed 
at  1  in  20,000  per  reactor  per 
year). 


Organizations  currently  using 
elicitation-based  risk  analysis 


■  Nuclear  Regulatory  Commission 

■  Department  of  Energy 


Environmental  Protection  Agency 
■  Department  of  Defense 


■  NASA 


■  1967  Apollo  flight  loss  -  spawned  one 

of  the  earliest  comprehensive  studies 


■  NUREG-1150  -  1991:  updated  PRA 
approach  based  on  Three-Mile 
Island,  and  improvements  in  risk 
assessment  research. 


■  1969  Goal:  Probability  of  loss  of  life 

<  1%  (space  shuttle  task  group) 


■  New  NRC  assessment:  will  include 
effect  of  emergency  preparedness 
and  other  mitigating  factors. 


■  1983  probabilistic  risk  analysis  of 

shuttle  flights:  NASA  administrators 
quickly  abandoned  PRA,  but  later 
events  proved  accuracy  of  analysis 


■  Intelligence  Community 

■  DHS:  Terrorism  Risk  Analysis 
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Expert  elicitation  of  event  probabilities 


Due  to  the  paucity  of  usable  terrorism 
data,  the  lack  of  justifiable  statistical 
methods  for  forecasting  threat,  and  the 
practical  challenges  of  conducting 
experimentally-designed  security 
evaluations,  alternative  means  must  be 
used  to  estimate  the  probability  of 
terrorism  scenarios  and  the  effectiveness 
of  homeland  security  programs. 

Statistics,  as  a  subject,  is  the... science  of 
handling  data.  On  the  other  hand 
probability,  as  a  subject,  we  might  say 
is  the  science  of  handling  the  lack  of 
data.  Thus,  one  often  hears  people  say 
that  we  cannot  use  probability  because 
we  have  insufficient  data... [but]  we  see 
that  this  is  a  misunderstanding.  When 
one  has  insufficient  data,  there  is 
nothing  else  one  can  do  but  use 
probability.  (Kaplan  and  Garrick,  1981) 


Elicitation  Protocol:  NUREG 1150 

Elicitation  approach  is  based  on  Bayesian 
theory  of  probability  and  on  decision 
analytic  models  and  techniques  for  eliciting 
and  using  expert  judgments.  The  overall 
process  for  eliciting  expert  judgments 
consists  of  the  following  steps: 

1)  Identification  and  selection  of  the  experts; 

2)  Training  in  probability  judgments; 

3)  Presentation  and  discussion  of  the 
uncertain  events  and  quantities; 

4)  Analysis  and  data  collection; 

5)  Presentation  and  discussion  of  the  results 
of  step  4; 

6)  Elicitation; 

7)  Analysis,  aggregation,  and  documentation. 
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Expert  elicitation  tool 


■  RMAhasaDHS 
internally  developed 
tool  based  in  Excel  +  R 
for  expert  elicitation  of 
Dirichlet  distributions. 

■  Other  tools  could  be 
used  (e.g.,  Crystal  Ball, 
@Risk,  Excalibur). 

■  The  key  for  any  of  these 
tools  is  to: 

■  capture  expert  rationale 
used  in  making 
judgments;  and 

■  characterize  uncertainty 
as  probability 
distributions. 
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What  is  the  estimated  relative  frequency  with  which  an  Intemaitonal  Terrorist  Group  would 
select  each  of  the  four  agents  being  considered? 


Notional 


Agent /Material 

Rank,  High  to 
Low  (1=Highest) 

Reiative  to  most 
iikeiy 

(most  iikeiy  =  1) 

Inverse  Ratios 

Probability 

K  Estimation 

Chemical  Agent  B 

1 

1 

1 

81.07% 

Most  Likely  Probabilty 

0.8107 

Chemcial  Agent  C 

2 

5 

0.2 

16.21% 

5th  Percentile 

0.2 

Chemical  Agent  A 

3 

30 

0.03333 

2.70% 

95th  Percentile 

0.9 

Chemical  Agent  D 

4 

10000 

0.0001 

0.01% 

Sum  of  Inverse  Ratios 

1.23 

Range  Calculation 

3 

Upper  Calculation 

39 

Key  Details  to  Keep  in  Mind 

Lower  Calculation 

1 

Only  consider  terrorism  attacks  on  the  U.S.  Homeland  (not  U.S.  interests  abroad). 

Time  frame  of  interest  is  2010-2012. 

Comment(s) 

Excel  Cell(s) 

Expert  has  a  lot  of  experience  with  Chemical  Agent  B. 
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Reviewing  results  with  experts 


■  The  Excel  +  R  Tool  also 
permits  showing  all  the 
elicited  data  to  experts: 

■  By  individual  (each 
expert’s  Dirichlet 
distribution  shown  as 
an  individual  box  plot); 

■  And  as  aggregated 
(yellow  box  plot 
summarizing  a  mixture 
distribution). 


■  The  discussion  of  results 
and  aggregation  should 
let  experts  remain 
anonymous  in  the  review 
process. 


Isotcpe  Selection 

Intern ationa I  Terrorist  Organizations 


Notional  output  from  an  expert  elicitation  on 
probability  of  an  agent  being  selected  for  use  in 
an  attack 
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Program  effectiveness  judgment  inputs 


Fault  Trees  map  DHS 
programs  and  non-DHS 
programs  to  specific  events  in 
order  to  estimate  event  failure 
probabilities:  in  the  example 
to  the  right,  this  is  the 
probability  that  the  programs 
in  blue  will  fail  to  stop  the 
entry  of  a  terrorist  team. 

Fault  trees  quantify  program 
impacts  on  event  likelihood  by 
modeling  how  programs  can 
act  individually  and  in  concert 
to  manage  risk. 

Subject  matter  experts  are 
elicited  for  their  judgments  of 
how  well  each  program 
performs  its  designated 
role(s),  such  as  detection  and 
interdiction. 


USOG 

m 


ICEOI 

rm 


Notional  Fault  Tree 
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Comparing  benefits  to  costs 


■  Accounting  for  uncertainty,  modeling  and  simulation  allows  for  the  estimation  of 
distributions  for  baseline  risk  and  reduced  risk  and  thus  gross  benefit. 

■  Even  if  the  regulation  is  cost-beneficial  at  the  mean,  the  left-hand  tail  of  the 
distribution  of  gross  benefits  may  overlap  with  the  distribution  of  costs. 

■  Thus  we  evaluate  the  chance  that  WHTI  is  not  cost-beneficial  by  computing  the 
probability  that  gross  risk  reduction  benefits  are  greater  than  or  equal  to  costs,  or 
the  probability  that  net  risk  reduction  benefits  are  greater  than  or  equal  to  zero. 
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Sensitivity  and  uncertainty  analysis 


■  In  prescribing  “Key  Elements  of  a  Regulatory  Analysis,”  0MB  Circular  A-4  states  that  a  good 
analysis  provides  the  results  of  formal  sensitivity  and  other  uncertainty  analyses. 

■  Modeling  and  simulation  of  quantitative  risk  analysis  allows  for  the  formal  inclusion  of  both: 

■  Sensitivity  analysis:  the  computation  of  the  effect  of  changes  in  input  values  or  assumption  on  the  outputs. 

■  Uncertainty  analysis:  the  computation  of  the  total  uncertainty  induced  in  the  output  by  quantified 
uncertainty  in  the  inputs  and  models. 

■  Failure  to  perform  systematic  sensitivity  and  uncertainty  analysis  leaves  both  analysts  and 
decision  makers  unable  to  judge  the  adequacy  of  the  analysis,  and  the  conclusions  reached. 


Sensitivity  Analysis:  Point  of  Entry  and  Failure  Probabilities 

-  Negative  a  Positive 

Entry  by  Air  POE 

I  ' 

Entry  by  Sea  POE 

Entry  by  Land  POE 

Entry  between  POEs 

WHTI  Effectiveness 

12%  9%  6%  3%  0%  3%  6%  9%  12% 

Percent  Change  in  Risk  Reduction 

Notional  Data 

uo 

jas 


$10 

S5 

SO 


Evaluating  Robustness  of  Conclusions  by  Air  POE  Probability 


2SK  20K  15% 

Varying  Probability  of  Air  POE 


Notional  Data 
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Simple  results  presentation 


Notional 

Data 


Benefit-Cost  Comparison:  Status  Quo,  WHTi  and 
Aiternative  Countermeasures  A  and  B 


$0  $20  $40  $60  $80  $100  $120  $140  $160 

■  Cost  ■  Marginal  Benefit 
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Strengths,  assumptions,  and  limitations 


■  Strengths 

■  Relies  on  expert-assessed  probabilities  and  consequences  rather  than  lay  people’s 
perceptions  of  risk  (as  distinct  from  public  valuations). 

■  Quantitatively  characterizes  all  input  and  output  uncertainties  to  allow  for  sensitivity  and 
other  uncertainty  analyses. 

■  Is  flexible  enough  to  support  relatively  granular  or  general  scenarios  for  subject  matter 
experts  to  think  about  when  providing  probabilistic  estimates  of  events,  program 
effectiveness,  and  consequence. 

■  Assumptions  and  limitations 

■  Though  public  valuation  of  fatality  risk  can  be  incorporated  via  a  willingness-to-pay- 
based  value-of-statistical  life,  economic  risk  is  not  as  readily  valued. 

■  Since  elicited  probabilities  are  assumed  to  reflect  terrorist  preferences  until  updated,  this 
approach  assumes  that  a  static,  point-in-time  assessment  of  current  risk  is  adequate  for 
long-term  rule-making;  this  is  also  true  of  alternative  game-theoretic  approaches. 

■  Though  OMB’s  Final  Information  Quality  Bulletin  for  Peer  Review  allows  for  national 
security  exemptions,  use  of  classified  intelligence  judgments  as  inputs  limits 
transparency  and  may  also  limit  the  ability  to  present  some  aggregated  results  publicly. 

■  Eliciting  effectiveness  from  programmatic  SMEs  may  yield  the  most  detailed  insights 
from  those  most  familiar  with  unique  operational  roles,  but  doing  so  creates  a  principal- 
agent  problem  and  presents  a  degree  of  moral  hazard  with  asymmetric  information. 
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Scale,  pace,  and  computational  requirements 


Scale  and  pace 

■  All  baseline  CBRNE  terrorism  event  consequences  and  probabilities  for  scenarios, 
incident  chains,  and  program  failure  have  already  been  assessed  as  part  of  DHS  BTRA, 
CTRA,  RNTRA,  ITRA,  and  RAPID  risk  assessments. 

■  Only  3  WHTI  program  failure  probability  distributions  and  7  countervailing  risk  POE 
distributions  would  need  to  be  elicited. 


■  At  this  reduced  scope,  elicitation  protocol  would  require  approximately  4  to  12  hours  of 
intelligence  analysts’  time  and  8  to  24  hours  of  program  SMEs’  time. 

WHTI-specific  computational  requirements 


■  Software  capable  of  Monte  Carlo  simulations  over  probability  distributions  (e.g.,  Matlab, 
Crystal  Ball); 

■  Software  and  programs  for  statistical  analysis  and  visualization  of  simulation  results 
(e.g..  Excel,  R,  Matlab);  and 

■  Database  software  (e.g..  Excel,  Access,  or  SQL). 


■  Note:  risk  analysis  for  existing  CBRNE  events  required  specialized  models  for  estimating 
the  human  and  economic  effects  of  various  terrorism  attacks,  as  well  as  software  and 
programs  for  modeling  induced  and  indirect  economic  effects  (e.g.,  IMPLAN  for  I/O 
modeling,  GAMS  for  CGE  modeling). 
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Using  Logic  Models  to  Assess  Security  Benefits 

Presenter:  Victoria  A.  Greenfield  (RAND) 

Authors:  Victoria  A.  Greenfield,  Henry  H.  Willis,  and  Tom  LaTourette  (RAND) 


RAND 


INFRASTRUCTURE,  SAFETY, 
AND  ENVIRONMENT 


Using  Logic  Models  to  Assess 
Security  Benefits 


Greenfield,  Willis,  and  LaTourrette 

November  29,  201 1 
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Project  Origins  and  Approach 

•  CBP  seeking  to  develop  methods  to  assess 
security  benefits  of  proposed  rules 

•  RAND  approach  uses  logic  model  to  estimate 
benefits  qualitatively  and,  as  far  as  possible, 
quantitatively 

-  Builds  on  available  information  about  program 
operations,  strategy,  and  responsibilities 

-  Does  not  result  in  point  estimate 


RAND  2 


Outline 


> Describe  Role  of  Logic  Model 

*  Develop  Illustrative  Logic  Model  for  Processing 
Information  and  Controlling  Entry 

*  Demonstrate  Application  to  WHTI 

*  Discuss  Conclusions 


RAND 
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0MB  Circular  A-4*  Provides  Guidance  for 
Assessing  Benefits  of  Rules 

*  Favors  quantification,  especialiy  monetization 

*  Allows  other  approaches  if  quantification  is  too  difficuit 
or  couid  be  misleading 

“When  important  benefits  and  costs  cannot  be  expressed  in 
monetary  units,  BCA  [benefit-cost  analysis]  is  less  useful,  and  it 
can  even  be  misleading....”  (p.  10) 

*  Outlines  key  eiements  of  regulatory  analysis  (pp.  2-3) 

-  Explain  links  between  rules,  actions,  and  expected  benefits 

-  Identify  “no  action”  or  “next  best”  baseline 

-  Identify  expected  undesirable  side  effects  and  ancillary  benefits 

*See  0MB,  Circular  A-4,  “Subject:  Regulatory  Analysis,”  September  17,  2003 

RAND  4 


Logic  Model  Can  Be  Used  to  Link  Rules, 
Actions,  and  Benefits* 


*  Articuiating  CBP’s  “program  story” — what,  why,  and 
who — as  foundation  for  analyticai  assessment 

*  Identifying  ways  in  which  proposed  regulation,  e.g., 
WHTI,  would  affect  program  story 

*  Assessing  regulatory-induced  changes  in 
capabilities  and  outcomes  in  terms  of  program  story 


*Logic  models  can  also  be  used  to  specify  performance  objectives  per  E.O.  13563; 
for  additional  information  on  potential  uses,  see,  e.g.,  Greenfield,  V.A.,  V.  Williams,  and  E. 
Eiseman,  Using  Logic  Models  for  Strategic  Planning  and  Evaluation,  Application  to  the 
National  Center  for  Injury  Prevention  and  Control,  TR-370-NCIPC,  RAND,  2006 
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Articulating  CBP's  Program  Story 


> Describe  operations 

-  What  are  you  doing? 

*  Describe  strategy 

-  Why  are  you  doing  it? 


Map  operations  to 
strategy 

Do  the  pieces — what 
and  why — fit  together? 


*  Delineate  responsibilities  and  boundaries 

-  l//fio  are  “you”? 

-  What  externai  factors  affect  your  performance? 

-  What  are  the  anciiiary  benefits  of  your  program? 

-  What  are  the  potentiai  side  effects  of  your  program? 

-  How  do  you  gauge  progress? 


RAND  6 


Describe  Operations 


Operations 


► 


“Customer”  represents  an  interface  and  might 
include  supported  or  partner  programs 


Operations  contribute  to  mission  attainment, 
both  sequentially  and  with  feedback 


t 


1 


1 


External  factors 


1 


j 


RAND 
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Describe  Strategy 


Operations  P 


Goals  and  objectives  derive  from  mission 

4  Strategy 


External  factors 


RAND 


Map  Operations  to  Strategy 


Operations 


► 


Program 

Inputs 

1  ,|  Program  |_J 

n  Activities  H 

Outputs 

(Inputs) 

1 

i 

Management  Annual 

Objectives  Goals 

Goals  and  objectives  relate  clearly  to  relevant 
operations,  e.g.,  strategic  goals  speak  to  outcomes 

strategy 


t 


1 


1 


External  factors 


1 


j 
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Delineate  Responsibilities  and  Boundaries 


Operations  ■  ► 


I  L 


External  factors 


J  L 


RAND 
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Specify  Performance  Measures 


Operations  ■  ¥ 


[ 


1 


1 


External  factors 


1 


j 
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CBP  Mission  Statement* 

•  We  are  the  guardians  of  our  Nation's  borders. 

•  We  are  America's  frontline 

•  We  safeguard  the  American  homeland  at  and 
beyond  our  borders 

•  We  protect  the  American  public  against  terrorists 
and  the  instruments  of  terror 

•  We  steadfastly  enforce  the  laws  of  the  United 
States  while  fostering  our  Nation's  economic 
security  through  lawful  international  trade  and  travel 

•  We  serve  the  American  public  with  vigilance,  integrity, 
and  professionalism 


Outcomes 


Safeguard  the 
American  homeland, 
protect  the  American 
public,  enforce  the  laws 
of  the  United  States 
while  fostering  our 
Nation's  economic 
security 


*See  CBP,  Secure  Borders,  Safe  Travel,  Legal  Trade:  U.S.  Customs  and  Border 
Protection  Fiscal  Year  2009-2014  Strategic  Plan,  p.  4,  July  2009 


External  factors 


RAND 


14 


Create,  populate, 
and  analyze 
databases 


Prevent  access  at 
border  and/or  prior 
to  attempted  entry 
Interface  with  other 
agencies,  internaiiy 
and  externaiiy 


t  L 


External  factors 


J  L 
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*  Create,  populate, 
and  analyze 
databases 


Prevent  access  at 
border  and/or  prior 
to  attempted  entry 
Interface  with  other 
agencies,  internally 
and  externally 


Undesirable 
side  effects 


1  L 


External  factors 


J  L 
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Outline 


*  Describe  Role  of  Logic  Model 

*  Develop  Illustrative  Logic  Modei  for  Processing 
information  and  Controlling  Entry 

> Demonstrate  Application  to  WHTi 

*  Discuss  Conclusions 


RAND 
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Border  Entry  Operations  Involve  Three  Steps 

*  Individual  approaches  border 

*  Individual  presents  identification  documents 

*  Customs  agent  obtains  documents* 

-  Verifies  authenticity 

-  Enters  information 

•  Checks  information  against  databases 

•  “Feeds”  databases 

-  Makes  decision  to  ailow  or  disallow  entry 

*Vicinity-read  radio-frequency  identification  chip  does  not  require  physicai  handiing 

RAND « 


WHTI  In  a  Nutshell 


•  Prior  to  WHTI,  regulations  permit  U.S.  citizens  and 
nonimmigrant  aliens  from  Canada,  Bermuda,  and 
Mexico  to  enter  United  States  without  passport 

*  Subsequent  to  WHTI,  regulations  require  that  nearly 
all  entrants  (land,  sea,  or  air)  present  either 

-  Passport  book 

-  Passport  card  with  vicinity-read  radio-frequency 
identification  chip 

-  CBP  trusted  traveller  card 

-  DHS  Enhanced  Driver’s  License 

-  Merchant  Mariner  Document 


RAND 
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How  Would  WHTI  Affect  Program  Story? 

*  WHTI  does  not  affect  structure  of  story  but  might 
affect  key  “parameters” 

*  Additional,  more  consistent  documentation 
requirements  might,  for  example 

-  Reduce  processing  time 

-  Improve  quantity  and/or  quaiity  of  information 

-  Increase  evidentiary  burden  on  traveiers  and  shippers 

*  WHTI  might,  uitimateiy,  affect  program  outcomes 
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How  Would  Changes  In 
Capabilities  Affect  Outcomes? 

*  Reduction  in  processing  time* 

-  Potential  for  reaiiocation  of  staff  and/or  underlying 
resources  to  meet  other  needs 

*  Improvement  in  quality/quantity  of  information 

-  Pofenf/a/ to  deter/prevent  improper  entry 

-  Potential  to  deter/prevent  other  undesirable  behavior, 
e.g.,  drug  and  human  trafficking 

*  Increase  in  evidentiary  burden 

-  Potential  to  interfere  with  normal  travel  and  commerce 

‘Also  potential  to  promote  normal  travel  and  commerce 


RAND 
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verifies  authenticity 

Customs  agent 
checks  information 
against  databases 

Customs  agent  makes 
decision  to  aiiow  or 
disaiiow  entry 


1  L 


External  factors 


J  L 
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WHTI  WHTI  WHTI 

WHTI  is,  itself,  an  external  factor 


RAND 
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WHTI  might  result  in  changes  in  capabilities  and  outcomes 
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Changes  Would  Occur  Along  Pipeline, 
but  Depend  on  Other  External  Factors 

*  Presence  and  sufficiency  of  infrastructure 

-  Checkpoint  facilities,  e.g.,  ratio  of  travellers  to  lanes 

-  information  technology 

*  Quaiity  and  availability  of  fake  documents 

*  Communication  between  CBP  and  other  U.S. 
government  and  foreign  officiais 


RAND 
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Logic  Model  Carries  Light  Technical  Burden 
but  Power  Depends  on  Quality  of  Data 

•  Approach  does  not  reiy  on 

-  Large  numbers  of  assumptions 

-  Complex  computing  or  statistical  methods 

*  Approach  is  more  convincing  with  good  data 

-  Estimating  benefits,  side  effects  along  pipeline 

-  Estimating  benefits,  side  effects  vis-a-vis  outcomes 


Approach  cannot  give  point  estimate 


RAND 


29 


Invited  Presentations  from  Workshop  Participants  61 


Outline 

*  Describe  Role  of  Logic  Model 

*  Develop  Illustrative  Logic  Model  for  Processing 
Information  and  Controlling  Entry 

*  Demonstrate  Application  to  WHTI 
>Discuss  Conclusions 

RAND  30 


Conclusions 

•  Application  of  tried-and-true  methodology  may 
advance  benefits  “estimation”  at  low  cost 

-  Logic  model  cannot  provide  single,  absolute  measure,  but 
can  improve  rationale  behind  regulatory  action 

-  Once  developed,  analytical  burden  of  use  is  modest 

*  Logic  model  provides  tool  for 

-  Articulating  CBP  program  story 

-  Identifying  ways  in  which  proposed  regulation  would 
impact  program  story 

-  Assessing  regulatory-induced  changes  in  capabilities  and 
outcomes  in  terms  of  program  story 

Approach  is  widely  applicable 


RAND 
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Abbreviations 


CBP 

U.S.  Customs  and  Border  Protection 

DHS 

U.S.  Department  of  Homeland  Security 

DOS 

U.S.  Department  of  State 

E.O. 

Executive  Order 

FBI 

U.S.  Federal  Bureau  of  Investigations 

0MB 

U.S.  Office  of  Management  and  Budget 

WHTI 

Western  Hemispheric  Travel  Initiative 

RAND 
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Characterizing  Benefits  of  Anti-Terrorism  Rules 
Presenter:  James  K.  Hammitt 

(Harvard  University  and  Toulouse  School  of  Economics) 


Characterizing  benefits  of 
anti-terrorism  rules 


James  K.  Hammitt 

Harvard  University  (Center  for  Risk  Analysis) 
Toulouse  School  of  Economics  (LERNA-INRA) 
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Purpose  of  regulatory  assessment 

Evaluate  if  rule  benefits  US  population 

-  Approximate  by  whether  total  benefits  to  population  exeeed 
total  costs 

Costs  (opportunity  cost  of  resources,  other  harms) 

-  Government 

•  Labor  (border  agents),  equipment  (e.g.,  sereening  deviees) 

-  Others  (e.g.,  eommereial  &  other  travelers) 

•  Time  waiting  for  inspeetions,  invasive  sereening  proeedures 

•  Restrietion  of  eivil  liberties 

Benefits 

-  Protection  from  terrorism 

Quantities  /  magnitudes  matter  -  question  of  tradeoffs 

2 


Benefits  =  reductions  in 

Harm  from  terrorist  attacks 

-  Deaths,  injuries,  destmetion  of  property,  disruption 

Precautionary  behaviors 

-  Labor  &  eapital  devoted  to  hardening  &  proteeting 
potential  targets 

-  Other  behavioral  response  (e.g.,  driving  rather  than 
flying,  avoiding  signature  events) 

Fear  &  anxiety 
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BCA  &  break-even  analysis  use  same 
information,  just  present  it  differently 

Assume  primary  benefit  is  reduction  in  harm  from  attacks 

A(P  •  L)  «  AP  •  L  [+  AL  •  P  +  AP  •  AL] 

Benefit-cost  analysis 

NB  =  A(P  •  L)  -  C  >  0 

•  AP,  AL,  C  are  uncertain 

•  Represent  uncertainty  by  probability  distributions,  calculate  probability 
distribution  of  NB 

•  Sensitivity  analysis:  for  what  values  of  AP,  AL,  C  are  NB  >  0? 

Break-even  analysis 

Assume  uncertainty  about  AP  much  larger  than  about  AL  and  C 
NB  =  AP  •  L  -  C  >  0 
AP  >  AP*  =  C  /  L 

•  Uncertainty  analysis:  calculate  probability  distribution  for  AP*  (for  which 
NB  is  positive) 

•  Sensitivity  analysis:  calculate  AP*  for  alternative  values  of  C,  L 


Transparency  v.  secrecy 

Some  information  should  be  withheld  from 
terrorists 

-  Details  of  protective  systems  at  targets,  entry 
portals 

Regulatory  assessments  cannot  include 
secret  information  (?) 

-  Limits  ability  of  public  &  others  to  evaluate  quality 

-  RMS  model 
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Estimating  benefits 

Harnn  fronn  terrorist  attacks 
Precautionary  behaviors 
Fear  &  anxiety 


Harm  from  terrorist  attacks 

Bottom  up  (damage  function) 

-  Estimate  deaths,  injuries,  property  destruction, 
disruption 

•  Scenarios  (discrete  cases)  or  probability 
distributions  (continuous  or  discrete) 

-  Value  elements  of  damages 
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Estimating  reductions  in 
probability  &  damages 

Intelligent  adversary 

-  Likely  to  respond  to  protective  actions  (those  it  knows 
or  can  discover) 

Protective  measures  (protecting  a  target  or  an 
access  route)  have  two  effects 

-  Make  attack  more  difficult 

•  Prevent  attacks 

-  Increase  relative  difficulty  of  target/route 

•  Divert  attacks 

Assessment  must  consider  countervailing  risk 
increases 

-  Make  decisions  strategically,  including  portfolio  effects 
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Estimating  reductions  in 
probability  &  damages 

Expert  judgment  is  essential 

-  Actuarial  estimates  inadequate  -  limited  data,  great 
heterogeneity  over  time  &  location,  conditions  change 
so  present  &  future  may  not  be  like  past 

-  RAND  database 

Subjective  probabilities 

-  Quantification  of  belief 

-  There  is  no  correct  value 

-  Experts  may  hold  different  probabilities 

-  Hard  to  evaluate  &  compare  quality 

-  Broad  range  of  values  may  be  legitimate 
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Estimating  benefits 

Precautionary  behaviors 

-  specific  to  regulation 

-  Model,  estimate  from  analogous  cases 

Fear  &  anxiety 

-  Not  necessarily  proportional  to  risk 

-  Placebos 

•  Effective  if  known? 

Both  types  depend  on  public  information, 
confidence  in  measures,  perceived  risks 

10 


Valuing  elements  of  damages 

Deaths  &  injuries 

-  Conventional  estimates  exist 

•  Limited  for  non-fatal  injuries 

-  Premia  for  ambiguity  or  catastrophe  aversion? 

•  Plausible  but  little  information  about  magnitude 

•  Literature  suggests  effect  much  smaller  than  order  of  magnitude 

Property  destruction 

-  Replacement  cost 

-  Premium  for  historically  or  culturally  significant  structures 

Disruption 

-  Bottom-up  model  of  time  &  productivity  losses 

•  Hard  to  model  interactions 

-  Top-down  analogy/extrapolation  from  effects  of  other 
catastrophic  events,  e.g.,  industrial  accident,  hurricanes, 
tornados,  earth-quakes 

•  Provide  insight  into  how  people  respond  ii 
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Valuing  elements  of  damages 

Precautionary  behaviors 

-  Demand  curves  for  travel  modes,  travel-cost 
methods,  engineering-cost  methods 

Anxiety  &  fear 

-  Stated  preference,  monetized  QALYs  (Adler) 
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Positive  V.  Normative  BCA? 

Conventional  BCA  includes  mix  of  individual/public  and 
expert/scientific  inputs 

-  Values  of  benefits  &  costs  based  on  individual  preferences 

-  "Objective"  assessment  of  risks  and  other  consequences 

Individual  behavior  and  perceptions  sometimes 
inconsistent  with  economic  model 

-  Cognitive  errors? 

-  Oversimplified  model? 

How  should  BCA  incorporate  departures  from  model? 

-  Populism  V.  paternalism? 

-  Role  of  government? 

13 
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Behavior  often  differs  from 
standard  economic  model 

Behavior  is  wrong? 

-  Cognitive  error  (susceptibility  to  framing,  excess 
attention  to  salient  attributes,  nonlinear  use  of 
probabilities) 

-  Self-control  problems  (procrastination) 

Model  is  wrong? 

-  Omits  important  attributes  (type  of  mortality  risk) 

-  Idealized  assumptions  (perfect  information  & 

processing)  14 


BCA  Includes  Two  Steps 

Predicting  consequences  of  alternative  policies 

-  Positive  question 

-  Predict  as  accurately  as  possible,  use  descriptively 
accurate  models 

•  Departures  from  standard  economic  model  may  be  appropriate 

Evaluating  consequences  of  alternative  policies 

-  Normative  question 

-  Use  consumers’  reflective,  informed  preferences 

-  How  can  these  be  determined? 

15 
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Improving  Estimation  of  the  Benefits  of  Terrorism  Risk  Reduction: 
Learning  from  Environmental  Economists? 

Presenter:  Carolyn  Kousky  (Resources  for  the  Future) 


I 

1  Improving  estimation  of  the  benefits 

1  of  terrorism  risk  reduction: 

1  learning  from  environmental  economists? 

w 

Carolyn  Kousky 
November  29,  2011 

RESOURCES 

FOR  THE  FUTURE 
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Overview  of  talk 

To  estimate  benefits  associated  with  reducing  terrorism  risk 
need: 

1)  to  know  how  risk  will  change  with  the  regulation 

1)  Estimating  baseline  risk 

2)  Estimating  how  baseline  risk  altered  by  regulation 

2)  the  value  of  that  risk  change 


Estimating  Changes  in  Risk 
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What  can  we  learn  from  climate  economics? 

►  Initial  approach  was  to  maximize  economic  efficiency 
using  Integrated  Assessment  Models 

►  emissions  temperature  economic  impacts 

►  Though  these  models  can  be  solved  to  yield  “optimal” 
solutions,  those  answers  are  dependent  on  the 
assumptions  and  biases  in  the  model  and  its 
paramaterization 

►  Initial  damage  functions  calibrated  from  small 
temperature  changes  and  limited  sectors 

►  But,  like  terrorism,  it  is  the  long  tails  that  matter! 

►  Get  a  huge  range  of  “answers”  based  on  model 
assumptions 


Aggregate  models  with  black  boxes 

►  Risk  =  probability  x  consequences 

Probability 

►  Estimate  probabilities  with  different  scenarios  for  population, 
economic  growth,  climate  sensitivity,  etc. 

►  Feedbacks/tipping  points  /climate  thresholds  difficult  to  model 
well 

►  Consequences 

►  Exposure  and  population  well  mapped 

►  But  some  threats  outside  realm  of  experience  and  these  could 
be  large  portion  of  losses  in  expected  value  terms 

►  Indirect  consequences  not  well  modeled  (cascading 
consequences;  interactions  &  feedbacks) 

►  Aggregation  not  easy  -cross-sectoral  interactions 
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Often  the  benefits  that  matter  are  local 


►  Multiple  climate  impacts  (multiple  types  of  attack) 

►  Multiple  locations  for  impact  (multiple  targets) 

►  Getting  to  local  impacts  from  abatement  policies 

►  Couple  downscaling  with  local  models 

►  For  WHTI,  aggregate  risk  reduction  might  be  OK;  other 
regs  will  be  more  targeted  at  types  of  attack  or  targets 

►  Need  to  explicitly  model  at  this  scale  or  back  out  from 
aggregate  down  to  local  impacts 


Getting  inside  the  black  boxes:  some  tools 
to  help  -  Structured  Expert  Judgment 

►  Experts  can  quantify  uncertainty  as  subjective  probability 

►  Treat  expert  judgments  as  scientific  data 

►  Use  when  there  are  theories  and  measurements  relevant  to  the  problem  but 
variables  of  interest  cannot  be  directly  measured  themselves 

►  Protocols  —  (e.g.,  Cooke  1991) 

Pre-Elicitation 

(1)  Definition  of  case  structure 

(2)  Identification  of  target  variables 

(3)  Identification  of  query  variables 

(4)  Identification  of  performance/seed/calibration  variables 

(5)  Identification  of  experts 

(6)  Selection  of  experts 

(7)  Definition  of  elicitation  format  document 

(8)  Dry  run  exercise 

(9)  Expert  training  session 
Elicitation 

(10)  Expert  elicitation  session  (individually) 

Post-Elicitation 

(11)  Combination  of  expert  assessments 

(12)  Discrepancy  and  robustness  analysis 

(13)  Feed  back 

(14)  Post-processing  analyses 

(15)  Documentation 

► 
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Getting  inside  the  black  boxes:  some  tools 
to  help  -  Structured  Expert  Judgment 

►  Defining  good  query  variables 

►  e.g.  not  exponent  in  arbitrary  damage  function 

►  Elicitation  format 

How  long  will  it  take  before  a  successful  terrorist  attack  on 
the  US  occurs  by  a  terrorist  who  enters  the  country  using 
false  documentation  at  a  Mexican  or  Canadian  crossing? 

Please  state  your  5%,  50%  and  95%  values,  taking  account  of: 

>  All  other  current  security  measures  remaining  unchanged 

>  Current ‘introduction  effort’  remaining  unchanged  (as  in 
invasive  species) 

►  Terrorists  adapt  strategies  to  the  new  situation  over  time 

►  Statistical  fluctuations 


► 

Getting  inside  the  black  boxes:  some  tools 
to  help  -  Structured  Expert  Judgment 

►  Need  to  identify  calibration  variables 

►  Example: 

In  London  2000,  weekly  average  PM ,0  was  18.4  pg/m^  What  is  the  ratio: 

#  non-accidental  deaths  in  the  week  with  the  highest  average  PM|o  concentration  (33.4  pg/m^) 
Weekly  average  #  non-accidental  deaths. 

5%  ; _ 25%; _  50%  ; _  75%; _ 95%; _ 

►  Can  use  these  to  do  better  than  equal  weighting  of 
experts 

>  Statistical  likelihood 

►  Are  the  expert’s  probability  statements  statistically  accurate? 

>■  Informativeness 

►  Probability  mass  concentrated  in  a  small  region,  relative  to 
background  measure  (e.g.  uniform  distribution  over  entire  range) 


► 
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Getting  inside  the  black  boxes:  some  tools 
to  help  -  Structured  Expert  Judgment 

►  Who  are  the  experts 

►  Nordhaus  1994  survey  found  natural  scientists  more  likely  to 
think  climate  change  could  lead  to  “severe  economic 
consequences”  and  economists  (non-environmental)  least 
likely. 

►  For  terrorism? 

►  Academics  studying  terrorism? 

►  Border  agents? 

►  CIA  analysts? 

►  EJ  can  be  costly  and  time  intensive  -  may  not  be  worth  it 

►  Cheaper  and  faster  if  sacrifice  some  quality 


Concerns  with  Structured  Expert  Judgment 

►  Some  opposition: 

►  Can  be  costly  and  time  intensive 

►  False  sense  of  precision 

►  Delay  in  regulatory  rule  making 

►  Lack  of  expertise  within  agency 

►  Remember:  SEJ  quantifies,  but  does  not  remove  uncertainty 

►  EPA  guidance  paper  on  use  of  EJ;  could  do  something 
similar  for  CBP 
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Getting  inside  the  black  boxes:  some  tools 
to  help  -  Precursors  and  Near  Misses 

►  Precursors;  conditions,  events,  and  sequences  that  precede  and  lead 
up  to  events  of  a  certain  severity 

►  Allows  for  more  data  to  estimate  the  probability  of  an  attack 

►  Requires  an  incident  reporting  system 

►  Can  be  used  to  reveal  unknown  failure  modes  or  for  trending  safety 
of  system.  Here  could  be  combined  with  other  tools  to  help  model 
relationship  between  regulations  and  risk  reduction. 

►  E.g.  Data  on  number  of  times  passport  scan  matches  positive  against 
terrorist  watch  list?  Number  of  people  investigated  further  based  on 
differing  documentation?  How  many  arrested  suspected  terrorists 
entered  US  from  Canada  or  Mexico? 

►  RMS  2008  reports  that  there  have  been  over  30  attack  plots  uncovered 
in  US  since  2001  (use  details  on  these  to  support  and  calibrate  model  - 
e.g.  revising  downward  likelihood  of  surface  to  air  missile  attack) 

►  Not  just  terrorist  activities,  but  trends  in  vulnerability  and  security 


Cascading  uncertainty 

►  Cascading  uncertainty  (Schneider  1983) 

►  Emissions^carbon  cycle  response^CS^ regional 
change^possible  impacts 

►  Need  uncertainty  analysis  not  just  within 
model,  but  across  models  (e.g.Kopp  etai  201 1) 
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Valuing  Changes  in  Risk 


Getting  to  WTP 

►  Want  to  estimate  willingness-to-pay  to  reduce  risk 

►  Risk  of  multiple  types  of  damage  (e.g.,  Hurricane  Katrina) 

►  Structural  damage,  including  productive  capital 

►  Indirect  economic  effects 

►  Loss  of  life  and  injury 

►  Some  impacts  can  be  estimated  in  dollars 

►  Models  of  structural  damage;  business  interruption 

►  Others  require  translating  metric,  e.g.  lives  lost,  to  dollar 
value 

►  Environmental  economists  have  long  history  of  this 


► 
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Fear  and  indirect  economic  impacts 

►  Behavioral  responses  can  play  a  large  role  (e.g.  rebuilding  post- 
Katrina) 

►  Fear  Actions  Economic  consequences 

►  More  research  needed  linking  level  of  fear  to  behavior 

►  Surveys  difficult  to  use;  studies  show  people  believe  their  DM  is  not 
driven  by  “visceral”  emotions  but  by  rational  deliberation  (Lowenstein, 
2000) 

Econometric  analyses  linking  tourism  to  terrorist  risk;  hedonic 
studies  linking  housing  prices  to  crime  rate;  investors  in  financial 
markets  (example:  Becker  and  Rubenstein  2011) 

►  Note  not  all  security  measures  reduce  fear  and  thus  increase 
economic  activity. 

►  More  straightforward  models  of  the  impact  of  changes  in 
demand,  say,  can  then  be  used  to  model  the  economic 
consequences 


A  presumption  of  risk  aversion 

►  Risk  neutrality  has  been  argued  as  not  appropriate  for 
climate  change  since  the  potential  impacts  are  global, 
damages  could  be  very  large,  and  correlation  among  risks 

undermines  any  risk  sharing  arrangement  (Newbold  and 
Daigneault  2010). 

►  Survey  evidence  that  people  are  more  risk  averse  for 
catastrophic  impacts,  large  loss  of  life,  or  serious  injury 

►  Similar  arguments  in  favor  of  a  risk  averse  position  could 
be  made  for  estimating  benefits  of  reducing  terrorism  risk 
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Risk  aversion  and  non-market  impacts 

►  Loss  of  life 

►  Injury 

►  Risk  averse  individuals  will  pay  more  than  ED  to  eliminate 
a  risk 

WTP  =  ED  +  RP 

►  There  is  a  value  to  changes  in  perceived  safety;  fear 


► 


Valuing  non-market  impacts 

1.  Revealed  Preference  Approaches 

Infer  value  of  a  good  from  market  transactions 

2.  Stated  Preference  Approaches 

Ask  people  hypothetical  questions 

3.  Model  utility  explicitly 

►  lAMs  assume  CRRA  utility 

►  Tangles  preferences 

►  Work  from  Weitzman  loosely  suggests  that  due  to  massive  uncertainties 
and  thick  tail,  WTP  for  abatement  could  swamp  everything  (and  with 
CRRA,  be  infinite) 

►  Caution  from  climate  work  is  that  utility  functions  that  work  well  in 
middle  part  of  distributions  might  not  work  well  at  all  at  extremes 

But  this  work  does  show  that  thinning  the  tail  can  be  enormously 
valuable  to  people 

4.  Put  more  weight  on  worst  case  outcomes 

► 
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RP:  Averting  behavior 

►  Private  purchases  to  reduce  a  risk  are  a  proxy  for  the 
value  of  risk  aversion 

►  E.g,:  water  filters,  safer  cars 

►  Can  be  difficult  to  tease  out  only  risk  reduction  because 
related  to  other  changes  in  the  good  or  may  produce 
other  benefits 


► 


CV  approaches 

►  Many  approaches  -  often  designed  as  referendum 

►  Study  design  matters  a  great  deal;  there  are  “best 
practices” 

►  Dealing  with  anomalies 

►  Anchoring;  protest  zeros;  not  adhering  to  budget  constraint 

►  Difficult  to  test  validity 


82  Assessing  the  Benefits  of  U.S.  Customs  and  Border  Protection  Regulatory  Actions  to  Reduce  Terrorism  Risks 


CV  to  value  reductions  in  terrorism  risk 

►  Embedding  effect  may  exist  (Viscusi  and  Zeckhauser  2003) 

►  What  is  the  specific  “commodity”  being  valued? 

►  Commodities  being  valued  should  be  as  decomposed  as 
matters  to  people’s  utility  -  but  still  need  to  be  able  to  identify 
substitution  and  complementarity 

►  Don’t  want  people  to  have  to  make  assumptions  about  links 
between  what  you  ask  and  what  they  care  about  -  could 
introduce  error 

►  Concern  about  “action  bias”  or  an  “emotion  premium” 

(Sunstein  and  Zeckhauser  2010) 

►  Divergence  between  actual  and  perceived  risk  reduction? 


► 


CV  to  value  reductions  in  fear 

People  willing  to  pay  to  feel  safer  or  to  be  less  fearful? 

►  Should  this  be  a  benefit  in  CBA?  (Adler  2004) 

►  Price  “fear  days”? 

►  Or  change  from  particular  regulation? 

►  Is  a  regulation  “fear  inert”? 

►  Isolating  value  of  reducing  fear? 

►  For  CV,  may  not  want  actual  changes  in  risk,  but  description  of  the 
visual  clues  people  would  have  that  risk  may  have  changed 

►  “Fear  entrepreneurs” 
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Be  careful  of  double  counting 

►  Damage  categories: 

►  Structural  damage 

►  Indirect  economic 

►  Loss  of  life 

►  Injury 

►  RP 

►  Using  wage  hedonics,  depending  on  context,  could  include 
both  of  last  two  categories  and  RP 

►  Does  the  VSL  already  include  concerns  about  fear? 

►  Averting  behavior  -  cost  of  smoke  alarm  (no  fear  value) 
and  improved  safety  in  car  (some  fear  value) 


Learning  from  Benefits  Transfer 

►  Tempting  to  use  benefits  transfer  when  little  work 
directly  on  terrorism 

►  Done  often  with  VSL 

►  But  lit  that  people  value  risks  differently  depending  on 
their  control,  fear  or  dread,  etc. 

►  With  terrorism,  deaths  clustered,  perception  of  risk  and 
absolute  level  of  risk  is  heterogeneous  in  population, 
dimensions  of  value  other  than  damage  and  lives  lost 
(national  pride;  fear)  (Viscusi  2009) 

►  Viscusi  found  respondents  value  reduction  in  terrorism  deaths 
more  highly  than  reduction  in  deaths  from  natural  disasters 
and  as  highly  as  traffic  deaths,  which  are  more  diffuse  and 
personal  risk  is  greater. 
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Is  quantification  and  valuation  of 
some  benefits  foolish? 


Cautionary  tale  for  CBA? 

►  When  include  range  of  plausible  assumptions  for  damage 
function,  utility  (risk  aversion),  as  well  as  unknowns  in 
climate  system,  get  enormous  range  of  answers. 

►  Studies  have  found  economic  justification  for  shockingly 
large  range  of  policies  from  doing  almost  nothing  to  fast 
and  aggressive  abatement. 

►  So  some  arguing  this  not  a  good  approach  for  problems 
with  deep  uncertainty. 
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Alternatives  to  CBA? 

►  Just  make  them  less  decisive 

►  Use  models  to  build  intuition;  do  not  take  numbers  literally. 

►  Use  models  to  make  assumptions  explicit. 

^  Robust  policies 

►  Win-win  polices 

►  Optimizing  under  a  risk  constraint 

►  This  may  be  where  economics  is  best  suited  -  determining 
least-cost  strategies  and  designing  policies  to  meet  targets 
(Ackerman  et  al  2009) 

►  How  do  you  identify  the  risk  constraint? 


On  the  other  hand... 

►  Lots  of  uncertainty  everywhere 

►  lAMs  do  let  you  ask  the  question:  what  beliefs  are 
required  for  benefits  to  be  large?  What  beliefs  are 
required  for  benefits  to  be  small? 

►  “Sometimes  the  best  that  can  be  done  is  to  specify  an 
exceedingly  wide  “benefits  range,”  one  that  does  not  do  a 
great  deal  to  discipline  judgment.”  But  CBA  can  still  be 

better  than  the  intuitive  judgments  people  make  -  (Sunstein 
2002) 

►  Large  literature  on  how  people  make  “bad”  or  “wrong” 
decisions  in  cases  of  risk  and  uncertainty  -  or  sometimes 
irrational  may  be  rational  after  all 
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Ancillary  benefits  to  research  approach? 

►  Some  of  the  methods  to  improve  the  benefit  estimate 
may  have  ancillary  benefits  in  terms  of  better  identifying 
the  most  effective  policies,  in  getting  policymakers  to  pay 
attention  to  the  most  likely  threats,  or  in  better 
identifying  and  integrating  public  preferences  in 
policymaking 


► 


Thank  you 

kousky@rff.org 
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